Imager with array of multiple infrared imaging modules

ABSTRACT

An imager array may be provided as part of an imaging system. The imager array may include a plurality of infrared imaging modules. Each infrared imaging module may include a plurality of infrared sensors associated with an optical element. The infrared imaging modules may be oriented, for example, substantially in a plane facing the same direction and configured to detect images from the same scene. Such images may be processed in accordance with various techniques to provide images of infrared radiation. The infrared imaging modules may include filters or lens coatings to selectively detect desired ranges of infrared radiation. Such arrangements of infrared imaging modules in an imager array may be used to advantageous effect in a variety of different applications.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 61/745,193 filed Dec. 21, 2012 and entitled “IMAGER WITHARRAY OF MULTIPLE INFRARED IMAGING MODULES” which is hereby incorporatedby reference in its entirety.

This application is a continuation-in-part of U.S. patent applicationSer. No. 13/043,123 filed Mar. 8, 2011 and entitled “IMAGER WITHMULTIPLE SENSOR ARRAYS”, which claims the benefit of U.S. ProvisionalPatent Application No. 61/312,146 filed Mar. 9, 2010 and entitled “MULTISPECTRAL MINIATURE SENSOR”, all of which are hereby incorporated byreference in their entirety.

This application is a continuation-in-part of U.S. patent applicationSer. No. 14/101,245 filed Dec. 9, 2013 and entitled “LOW POWER AND SMALLFORM FACTOR INFRARED IMAGING” which is hereby incorporated by referencein its entirety.

U.S. patent application Ser. No. 14/101,245 is a continuation ofInternational Patent Application No. PCT/US2012/041744 filed Jun. 8,2012 and entitled “LOW POWER AND SMALL FORM FACTOR INFRARED IMAGING”which is hereby incorporated by reference in its entirety.

International Patent Application No. PCT/US2012/041744 claims thebenefit of U.S. Provisional Patent Application No. 61/656,889 filed Jun.7, 2012 and entitled “LOW POWER AND SMALL FORM FACTOR INFRARED IMAGING”which is hereby incorporated by reference in its entirety.

International Patent Application No. PCT/US2012/041744 claims thebenefit of U.S. Provisional Patent Application No. 61/545,056 filed Oct.7, 2011 and entitled “NON-UNIFORMITY CORRECTION TECHNIQUES FOR INFRAREDIMAGING DEVICES” which is hereby incorporated by reference in itsentirety.

International Patent Application No. PCT/US2012/041744 claims thebenefit of U.S. Provisional Patent Application No. 61/495,873 filed Jun.10, 2011 and entitled “INFRARED CAMERA PACKAGING SYSTEMS AND METHODS”which is hereby incorporated by reference in its entirety.

International Patent Application No. PCT/US2012/041744 claims thebenefit of U.S. Provisional Patent Application No. 61/495,879 filed Jun.10, 2011 and entitled “INFRARED CAMERA SYSTEM ARCHITECTURES” which ishereby incorporated by reference in its entirety.

International Patent Application No. PCT/US2012/041744 claims thebenefit of U.S. Provisional Patent Application No. 61/495,888 filed Jun.10, 2011 and entitled “INFRARED CAMERA CALIBRATION TECHNIQUES” which ishereby incorporated by reference in its entirety.

This application is a continuation-in-part of U.S. patent applicationSer. No. 14/099,818 filed Dec. 6, 2013 and entitled “NON-UNIFORMITYCORRECTION TECHNIQUES FOR INFRARED IMAGING DEVICES” which is herebyincorporated by reference in its entirety.

U.S. patent application Ser. No. 14/099,818 is a continuation ofInternational Patent Application No. PCT/US2012/041749 filed Jun. 8,2012 and entitled “NON-UNIFORMITY CORRECTION TECHNIQUES FOR INFRAREDIMAGING DEVICES” which is hereby incorporated by reference in itsentirety.

International Patent Application No. PCT/US2012/041749 claims thebenefit of U.S. Provisional Patent Application No. 61/545,056 filed Oct.7, 2011 and entitled “NON-UNIFORMITY CORRECTION TECHNIQUES FOR INFRAREDIMAGING DEVICES” which is hereby incorporated by reference in itsentirety.

International Patent Application No. PCT/US2012/041749 claims thebenefit of U.S. Provisional Patent Application No. 61/495,873 filed Jun.10, 2011 and entitled “INFRARED CAMERA PACKAGING SYSTEMS AND METHODS”which is hereby incorporated by reference in its entirety.

International Patent Application No. PCT/US2012/041749 claims thebenefit of U.S. Provisional. Patent Application No. 61/495,879 filedJun. 10, 2011 and entitled “INFRARED CAMERA SYSTEM ARCHITECTURES” whichis hereby incorporated by reference in its entirety.

International Patent Application No. PCT/US2012/041749 claims thebenefit of U.S. Provisional Patent Application No. 61/495,888 filed Jun.10, 2011 and entitled “INFRARED CAMERA CALIBRATION TECHNIQUES” which ishereby incorporated by reference in its entirety.

This application is a continuation-in-part of U.S. patent applicationSer. No. 14/101,258 filed Dec. 9, 2013 and entitled “INFRARED CAMERASYSTEM ARCHITECTURES” which is hereby incorporated by reference in itsentirety.

U.S. patent application Ser. No. 14/101,258 is a continuation ofInternational Patent Application No. PCT/US2012/041739 filed Jun. 8,2012 and entitled “INFRARED CAMERA SYSTEM ARCHITECTURES” which is herebyincorporated by reference in its entirety.

International Patent Application No. PCT/US2012/041739 claims thebenefit of U.S. Provisional Patent Application No. 61/495,873 filed Jun.10, 2011 and entitled “INFRARED CAMERA PACKAGING SYSTEMS AND METHODS”which is hereby incorporated by reference in its entirety.

International Patent Application No. PCT/US2012/041739 claims thebenefit of U.S. Provisional Patent Application No. 61/495,879 filed Jun.10, 2011 and entitled “INFRARED CAMERA SYSTEM ARCHITECTURES” which ishereby incorporated by reference in its entirety.

International Patent Application No. PCT/US2012/041739 claims thebenefit of U.S. Provisional Patent Application No. 61/495,888 filed Jun.10, 2011 and entitled “INFRARED CAMERA CALIBRATION TECHNIQUES” which ishereby incorporated by reference in its entirety.

This patent application is a continuation-in-part of U.S. patentapplication Ser. No. 13/437,645 filed Apr. 2, 2012 and entitled“INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITH FUSION” which ishereby incorporated by reference in its entirety.

U.S. patent application Ser. No. 13/437,645 is a continuation-in-part ofU.S. patent application Ser. No. 13/105,765 filed May 11, 2011 andentitled “INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITH FUSION”which is hereby incorporated by reference in its entirety.

U.S. patent application Ser. No. 13/437,645 also claims the benefit ofU.S. Provisional Patent Application No. 61/473,207 filed Apr. 8, 2011and entitled “INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITH FUSION”which is hereby incorporated by reference in its entirety.

U.S. patent application Ser. No. 13/437,645 is also acontinuation-in-part of U.S. patent application Ser. No. 12/766,739filed Apr. 23, 2010 and entitled “INFRARED RESOLUTION AND CONTRASTENHANCEMENT WITH FUSION” which is hereby incorporated by reference inits entirety.

U.S. patent application Ser. No. 13/105,765 is a continuation ofInternational Patent Application No. PCT/EP2011/056432 filed Apr. 21,2011 and entitled “INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITHFUSION” which is hereby incorporated by reference in its entirety.

U.S. patent application Ser. No. 13/105,765 is also acontinuation-in-part of U.S. patent application Ser. No. 12/766,739which is hereby incorporated by reference in its entirety.

International Patent Application No. PCT/EP2011/056432 is acontinuation-in-part of U.S. patent application Ser. No. 12/766,739which is hereby incorporated by reference in its entirety.

International Patent Application No. PCT/EP2011/056432 also claims thebenefit of U.S. Provisional Patent Application No. 61/473,207 which ishereby incorporated by reference in its entirety.

This application claims the benefit of U.S. Provisional PatentApplication No. 61/748,018 filed Dec. 31, 2012 and entitled “COMPACTMULTI-SPECTRUM IMAGING WITH FUSION” which is hereby incorporated byreference in its entirety.

This application is a continuation-in-part of U.S. patent applicationSer. No. 12/477,828 filed Jun. 3, 2009 and entitled “INFRARED CAMERASYSTEMS AND METHODS FOR DUAL SENSOR APPLICATIONS” which is herebyincorporated by reference in its entirety.

This application claims the benefit of U.S. Provisional PatentApplication No. 61/792,582 filed Mar. 15, 2013 and entitled “TIME SPACEDINFRARED IMAGE ENHANCEMENT” which is hereby incorporated by reference inits entirety.

This application claims the benefit of U.S. Provisional PatentApplication No. 61/793,952 filed Mar. 15, 2013 and entitled “INFRAREDIMAGING ENHANCEMENT WITH FUSION” which is hereby incorporated byreference in its entirety.

This application claims the benefit of U.S. Provisional PatentApplication No. 61/746,069 filed Dec. 26, 2012 and entitled “TIME SPACEDINFRARED IMAGE ENHANCEMENT” which is hereby incorporated by reference inits entirety.

This application claims the benefit of U.S. Provisional PatentApplication No. 61/746,074 filed Dec. 26, 2012 and entitled “INFRAREDIMAGING ENHANCEMENT WITH FUSION” which is hereby incorporated byreference in its entirety.

TECHNICAL FIELD

One or more embodiments of the invention relate generally to imagesensors and more particularly, for example, to devices with multiplesets of image sensors, infrared sensors, and associated optics.

BACKGROUND

There are a wide variety of image detectors, such as visible light imagedetectors, infrared image detectors, or other types of image detectorsthat may be used to capture images for storage and display. Recentadvances in process technology for focal plane arrays (FPAs) and imageprocessing have led to increased capabilities and sophistication ofresulting imaging systems. While these developments may provide improvedfeatures and image quality, they often negatively impact the size,weight, and power of associated systems. In particular, single apertureoptical systems supporting multispectral imaging are typically complex,heavy, and expensive. In addition, conventional high resolution sensorsin the long wave infrared band may require very large apertures.

As an example, thermal cameras are used frequently in military andcommercial applications. However, in many circumstances, size and weightlimitations may render such thermal cameras unpractical. Cost is also anobstacle for market penetration in many applications. In particular,infrared camera optics often contribute significantly to the total costand size of such devices. As a result, there is a need for improvedimage detector implementations that provide high capability withoutundue impact on size, weight, and power of image detectors or theirassociated devices.

SUMMARY

In accordance with various embodiments of the present disclosure, animager array may be provided as part of an imaging system. The imagerarray may include a plurality of infrared imaging modules. Each infraredimaging module may include a plurality of infrared sensors associatedwith an optical element. The infrared imaging modules may be oriented,for example, substantially in a plane facing the same direction andconfigured to detect multiple images from the same scene using theimager array. Such images may be processed in accordance with varioustechniques to provide images of infrared radiation. In some embodiments,the infrared imaging modules may include filters or lens coatings toselectively detect desired ranges of infrared radiation. Sucharrangements of infrared imaging modules in an imager array may be usedto advantageous effect in a variety of different applications asdescribed herein.

In accordance with an embodiment of the disclosure, an imaging systemincludes a system housing and an imager array disposed in the systemhousing and adapted to image a scene. The imager array may include aplurality of infrared imaging modules. Each infrared imaging module mayinclude a module housing, an optical element fixed relative to themodule housing and adapted to receive infrared radiation from the scene,and a plurality of infrared sensors in a focal plane array (FPA) adaptedto capture an image of the scene based on the infrared radiationreceived through the optical element.

In accordance with another embodiment of the disclosure, a method ofimaging includes receiving infrared radiation from a scene at an imagerarray disposed in a system housing of an imaging system. The imagerarray may include a plurality of infrared imaging modules. Each infraredimaging module may include a module housing, an optical element fixedrelative to the module housing and adapted to receive the infraredradiation from the scene, and a plurality of infrared sensors in an FPAadapted to capture an image of the scene based on the infrared radiationreceived through the optical element. The method may further includecapturing a plurality of images of the scene substantiallysimultaneously using the infrared sensors of the infrared imagingmodules.

In accordance with another embodiment of the disclosure, a gas detectionsystem includes an imager array adapted to image a scene, where theimager array may include a plurality of infrared imaging modules. Eachinfrared imaging module may include a module housing, an optical elementfixed relative to the module housing and adapted to receive the infraredradiation from the scene, and a plurality of infrared sensors in an FPAadapted to capture an image of the scene based on the infrared radiationreceived through the optical element. A first one of the infraredimaging modules may be adapted to capture a first image of a firstwavelength range of the infrared radiation, a second one of the infraredimaging module may be adapted to capture a second image of a secondwavelength range of the infrared radiation, and the second wavelengthrange may be a subset of the first wavelength range and substantiallycorrespond to an absorption band of a gas.

In accordance with another embodiment of the disclosure, a method ofdetecting gas includes receiving infrared radiation from a scene at animager array, wherein the imager array includes a plurality of infraredimaging modules. Each infrared imaging module may include a modulehousing, an optical element fixed relative to the module housing andadapted to receive the infrared radiation from the scene, and aplurality of infrared sensors in an FPA adapted to capture an image ofthe scene based on the infrared radiation received through the opticalelement. The method may further include capturing a first image of afirst wavelength range of the infrared radiation using a first one ofthe infrared imaging modules, and capturing a second image of a secondwavelength range of the infrared radiation using a second one of theinfrared imaging modules, where the second wavelength range may be asubset of the first wavelength range and substantially corresponds to anabsorption band of a gas.

In accordance with another embodiment of the disclosure, an imagercalibration system includes an imager array adapted to image a scene,wherein the imager array includes a plurality of infrared imagingmodules. Each infrared imaging modules may include a module housing, anoptical element fixed relative to the module housing and adapted toreceive the infrared radiation from the scene, a plurality of infraredsensors in an FPA adapted to capture an image of the scene based on theinfrared radiation received through the optical element, and aprocessor. The processor may be adapted to receive a plurality of pixelvalues associated with the images captured by the infrared sensors ofthe infrared imaging modules, to map the sensors to a coordinate space,where at least one infrared sensor of each infrared imaging modules ismapped to each coordinate of the coordinate space, and to calculate anoffset correction term for each infrared sensor based on the pixelvalues of all infrared sensors mapped to the same coordinate.

In accordance with another embodiment of the disclosure, a method ofcalibrating an imaging system includes receiving infrared radiation froma scene at an imager array, where the imager array includes a pluralityof infrared imaging modules. Each infrared imaging module may include amodule housing, an optical element fixed relative to the module housingand adapted to receive the infrared radiation from the scene, and aplurality of infrared sensors in an FPA adapted to capture an image ofthe scene based on the infrared radiation received through the opticalelement. The method may further include receiving a plurality of pixelvalues associated with the images captured by the infrared sensors ofthe infrared imaging modules, mapping the infrared sensors to acoordinate space, where at least one infrared sensor of each infraredimaging module is mapped to each coordinate of the coordinate space, andcalculating an offset correction term for each infrared sensor based onthe pixel values of all infrared sensors mapped to the same coordinate.

The scope of the invention is defined by the claims, which areincorporated into this section by reference. A more completeunderstanding of embodiments of the invention will be afforded to thoseskilled in the art, as well as a realization of additional advantagesthereof, by a consideration of the following detailed description of oneor more embodiments. Reference will be made to the appended sheets ofdrawings that will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an imaging system in accordancewith an embodiment of the disclosure.

FIGS. 2A-B illustrate several views of an imager array having aplurality of sensor arrays of substantially equal size in accordancewith embodiments of the disclosure.

FIGS. 2C-D illustrate several views of an imager array having aplurality of sensor arrays of different sizes in accordance withembodiments of the disclosure.

FIGS. 2E-F identify diameters of airy discs as a function of wavelengthin accordance with embodiments of the disclosure.

FIG. 3 illustrates a process of obtaining an image having a desirablesignal to noise ratio in accordance with an embodiment of thedisclosure.

FIG. 4 illustrates a process of obtaining a high resolution image usingphase shifts between sensor arrays in accordance with an embodiment ofthe disclosure.

FIG. 5 illustrates an imager array configured to provide stereo imagingin accordance with an embodiment of the disclosure.

FIG. 6A illustrates a process of correcting defective pixels in animager array in accordance with an embodiment of the disclosure.

FIGS. 6B-C illustrate images with defective pixels in accordance withembodiments of the disclosure.

FIGS. 7A-B illustrate processes of calibrating sensors of an imagerarray in accordance with embodiments of the disclosure.

FIG. 8A illustrates transmission as a function of wavelength for a gasthat may be detected by an imager array in accordance with an embodimentof the disclosure.

FIG. 8B illustrates transmission through the atmosphere as a function ofwavelength for an atmospheric condition that may be detected by animager array in accordance with an embodiment of the disclosure.

FIG. 8C illustrates a process of performing gas detection in accordancewith an embodiment, of the disclosure.

FIG. 9A illustrates an imager array including a plurality of sensorarrays and a beamsplitter in accordance with an embodiment of thedisclosure.

FIG. 9B illustrates an imager array including a plurality of cameras inaccordance with an embodiment of the disclosure.

FIG. 10 illustrates a process of providing a high resolution image usingan artificial neural network in accordance with an embodiment of thedisclosure.

FIGS. 11A-F illustrate several views and types of imager arrays having aplurality of infrared imaging modules in accordance with embodiments ofthe disclosure.

FIG. 12 illustrates an infrared imaging module configured to beimplemented in a host device in accordance with an embodiment of thedisclosure.

FIG. 13 illustrates an assembled infrared imaging module in accordancewith an embodiment of the disclosure.

FIG. 14 illustrates an exploded view of an infrared imaging modulejuxtaposed over a socket in accordance with an embodiment of thedisclosure.

FIG. 15 illustrates a block diagram of an infrared sensor assemblyincluding an array of infrared sensors in accordance with an embodimentof the disclosure.

FIG. 16 illustrates a flow diagram of various operations to determinenon-uniformity correction (NUC) terms in accordance with an embodimentof the disclosure.

FIG. 17 illustrates differences between neighboring pixels in accordancewith an embodiment of the disclosure.

FIG. 18 illustrates a flat field correction technique in accordance withan embodiment of the disclosure.

FIG. 19 illustrates various image processing techniques of FIG. 16 andother operations applied in an image processing pipeline in accordancewith an embodiment of the disclosure.

FIG. 20 illustrates a temporal noise reduction process in accordancewith an embodiment of the disclosure.

FIG. 21 illustrates particular implementation details of severalprocesses of the image processing pipeline of FIG. 19 in accordance withan embodiment of the disclosure.

FIG. 22 illustrates spatially correlated fixed pattern noise (FPN) in aneighborhood of pixels in accordance with an embodiment of thedisclosure,

FIG. 23 illustrates a block diagram of another implementation of aninfrared sensor assembly including an array of infrared sensors and alow-dropout regulator in accordance with an embodiment of thedisclosure.

FIG. 24 illustrates a circuit diagram of a portion of the infraredsensor assembly of FIG. 23 in accordance with an embodiment of thedisclosure.

Embodiments of the invention and their advantages are best understood byreferring to the detailed description that follows. It should beappreciated that like reference numerals are used to identify likeelements illustrated in one or more of the figures.

DETAILED DESCRIPTION

FIG. 1 illustrates a block diagram of an imaging system 100 inaccordance with an embodiment of the disclosure. Imaging system 100 maybe used to capture and process images in accordance with varioustechniques described herein. As shown, various components of imagingsystem 100 may be provided in a housing 101, such as a housing of acamera or other system. In one embodiment, imaging system 100 includes aprocessing component 110, a memory component 120, an image capturecomponent 130 (e.g., an imager array including a plurality of sensorarrays), a display component 140, a control component 150, and a modesensing component 160. In another embodiment, imaging system 100 mayalso include a communication component 152 and one or more other sensingcomponents 162.

In various embodiments, imaging system 100 may represent an imagingdevice, such as a camera, to capture images, for example, of a scene170. Imaging system 100 may represent any type of camera system which,for example, detects electromagnetic radiation and providesrepresentative data (e.g., one or more still images or video images).For example, imaging system 100 may represent a camera that is directedto detect one or more ranges of electromagnetic radiation and provideassociated image data. Imaging system 100 may include a portable deviceand may be implemented, for example, as a handheld device and/orcoupled, in other examples, to various types of vehicles (e.g., aland-based vehicle, a watercraft, an aircraft, a spacecraft, or othervehicle) or to various types of fixed locations (e.g., a home securitymount, a campsite or outdoors mount, or other location) via one or moretypes of mounts. In still another example, imaging system 100 may beintegrated as part of a non-mobile installation to provide images to bestored and/or displayed.

Processing component 110 includes, in one embodiment, a microprocessor,a single-core processor, a multi-core processor, a microcontroller, alogic device (e.g., a programmable logic device configured to performprocessing functions), a digital signal processing (DSP) device, or anyother type of generally known processor. Processing component 110 isadapted to interface and communicate with components 120, 130, 140, 150,160, and 162 to perform method and processing steps as described herein.Processing component 110 may include one or more mode modules 112A-112Nfor operating in one or more modes of operation (e.g., to operate inaccordance with any of the various embodiments disclosed herein). In oneaspect, mode modules 112A-112N are adapted to define preset processingand/or display functions that may be embedded in processing component110 or stored on memory component 120 for access and execution byprocessing component 110. In another aspect, processing component 110may be adapted to perform various types of image processing algorithmsas described herein.

In various embodiments, it should be appreciated that each mode module112A-112N may be integrated in software and/or hardware as part ofprocessing component 110, or code (e.g., software or configuration data)for each mode of operation associated with each mode module 112A-112N,which may be stored in memory component 120. Embodiments of mode modules112A-112N (i.e., modes of operation) disclosed herein may be stored by aseparate machine readable medium (e.g., a memory, such as a hard drive,a compact disk, a digital video disk, or a flash memory) to be executedby a computer (e.g., logic or processor-based system) to perform variousmethods disclosed herein.

In one example, the machine readable medium may be portable and/orlocated separate from imaging system 100, with stored mode modules112A-112N provided to imaging system 100 by coupling the machinereadable medium to imaging system 100 and/or by imaging system 100downloading (e.g., via a wired or wireless link) the mode modules112A-112N from the machine readable medium (e.g., containing thenon-transitory information). In various embodiments, as describedherein, mode modules 112A-112N provide for improved camera processingtechniques for real time applications, wherein a user or operator maychange the mode of operation depending on a particular application, suchas a off-road application, a maritime application, an aircraftapplication, a space application, or other application.

Memory component 120 includes, in one embodiment, one or more memorydevices to store data and information. The one or more memory devicesmay include various types of memory including volatile and non-volatilememory devices, such as RAM (Random Access Memory), ROM (Read-OnlyMemory), EEPROM (Electrically-Erasable Read-Only Memory), flash memory,or other types of memory. In one embodiment, processing component 110 isadapted to execute software stored in memory component 120 to performvarious methods, processes, and modes of operations in manner asdescribed herein.

Image capture component 130 includes, in one embodiment, one or moresensors (e.g., any type of detector, such as a focal plane array) forcapturing image signals representative of an image, of scene 170. In oneembodiment, the sensors of image capture component 130 provide forrepresenting (e.g., converting) a captured image signal of scene 170 asdigital data (e.g., via an analog-to-digital converter included as partof the sensor or separate from the sensor as part of imaging system100). Processing component 110 may be adapted to receive image signalsfrom image capture component 130, process image signals (e.g., toprovide processed image data), store image signals or image data inmemory component 120, and/or retrieve stored image signals from memorycomponent 120. Processing component 110 may be adapted to process imagesignals stored in memory component 120 to provide image data (e.g.,captured and/or processed image data) to display component 140 forviewing by a user.

Display component 140 includes, in one embodiment, an image displaydevice (e.g., a liquid crystal display (LCD)) or various other types ofgenerally known video displays or monitors. Processing component 110 maybe adapted to display image data and information on display component140. Processing component 110 may be adapted to retrieve image data andinformation from memory component 120 and display any retrieved imagedata and information on display component 140. Display component 140 mayinclude display electronics, which may be utilized by processingcomponent 110 to display image data and information. Display component140 may receive image data and information directly from image capturecomponent 130 via processing component 110, or the image data andinformation may be transferred from memory component 120 via processingcomponent 110.

In one embodiment, processing component 110 may initially process acaptured image and present a processed image in one mode, correspondingto mode modules 112A-112N, and then upon user input to control component150, processing component 110 may switch the current mode to a differentmode for viewing the processed image on display component 140 in thedifferent mode. This switching may be referred to as applying the cameraprocessing techniques of mode modules 112A-112N for real timeapplications, wherein a user or operator may change the mode whileviewing an image on display component 140 based on user input to controlcomponent 150. In various aspects, display component 140 may be remotelypositioned, and processing component 110 may be adapted to remotelydisplay image data and information on display component 140 via wired orwireless communication with display component 140, as described herein.

Control component 150 includes, in one embodiment, a user input and/orinterface device having one or more user actuated components, such asone or more push buttons, slide bars, rotatable knobs or a keyboard,that are adapted to generate one or more user actuated input controlsignals. Control component 150 may be adapted to be integrated as partof display component 140 to function as both a user input device and adisplay device, such as, for example, a touch screen device adapted toreceive input signals from a user touching different parts of thedisplay screen. Processing component 110 may be adapted to sense controlinput signals from control component 150 and respond to any sensedcontrol input signals received therefrom.

Control component 150 may include, in one embodiment, a control panelunit (e.g., a wired or wireless handheld control unit) having one ormore user-activated mechanisms (e.g., buttons, knobs, sliders, orothers) adapted to interface with a user and receive user input controlsignals. In various embodiments, the one or more user-activatedmechanisms of the control panel unit may be utilized to select betweenthe various modes of operation, as described herein in reference to modemodules 112A-112N. In other embodiments, it should be appreciated thatthe control panel unit may be adapted to include one or more otheruser-activated mechanisms to provide various other control functions ofimaging system 100, such as auto-focus, menu enable and selection, fieldof view (FoV), brightness, contrast, gain, offset, spatial, temporal,and/or various other features and/or parameters. In still otherembodiments, a variable gain signal may be adjusted by the user oroperator based on a selected mode of operation.

In another embodiment, control component 150 may include a graphicaluser interface (GUI), which may be integrated as part of displaycomponent 140 (e.g., a user actuated touch screen), having one or moreimages of the user-activated mechanisms (e.g., buttons, knobs, sliders,or others), which are adapted to interface with a user and receive userinput control signals via the display component 140. As an example forone or more embodiments as discussed further herein, display component140 and control component 150 may represent a smart phone, a tablet, apersonal digital assistant (e.g., a wireless, mobile device), a laptopcomputer, a desktop computer, or other type of device.

Mode sensing component 160 includes, in one embodiment, an applicationsensor adapted to automatically sense a mode of operation, depending onthe sensed application (e.g., intended use or implementation), andprovide related information to processing component 110. In variousembodiments, the application sensor may include a mechanical triggeringmechanism (e.g., a clamp, clip, hook, switch, push-button, or others),an electronic triggering mechanism (e.g., an electronic switch,push-button, electrical signal, electrical connection, or others), anelectro-mechanical triggering mechanism, an electro-magnetic triggeringmechanism, or some combination thereof. For example for one or moreembodiments, mode sensing component 160 senses a mode of operationcorresponding to the imaging system's 100 intended application based onthe type of mount (e.g., accessory or fixture) to which a user hascoupled the imaging system 100 (e.g., image capture component 130).Alternatively, the mode of operation may be provided via controlcomponent 150 by a user of imaging system 100 (e.g., wirelessly viadisplay component 140 having a touch screen or other user inputrepresenting control component 150).

Furthermore in accordance with one or more embodiments, a default modeof operation may be provided, such as for example when mode sensingcomponent 160 does not sense a particular mode of operation (e.g., nomount sensed or user selection provided). For example, imaging system100 may be used in a freeform mode (e.g., handheld with no mount) andthe default mode of operation may be set to handheld operation, with theimages provided wirelessly to a wireless display (e.g., another handhelddevice with a display, such as a smart phone, or to a vehicle'sdisplay).

Mode sensing component 160, in one embodiment, may include a mechanicallocking mechanism adapted to secure the imaging system 100 to a vehicleor part thereof and may include a sensor adapted to provide a sensingsignal to processing component 110 when the imaging system 100 ismounted and/or secured to the vehicle. Mode sensing component 160, inone embodiment, may be adapted to receive an electrical signal and/orsense an electrical connection type and/or mechanical mount type andprovide a sensing signal to processing component 110. Alternatively orin addition, as discussed herein for one or more embodiments, a user mayprovide a user input via control component 150 (e.g., a wireless touchscreen of display component 140) to designate the desired mode (e.g.,application) of imaging system 100.

Processing component 110 may be adapted to communicate with mode sensingcomponent 160 (e.g., by receiving sensor information from mode sensingcomponent 160) and image capture component 130 (e.g., by receiving dataand information from image capture component 130 and providing and/orreceiving command, control, and/or other information to and/or fromother components of imaging system 100).

In various embodiments, mode sensing component 160 may be adapted toprovide data and information relating to system applications including ahandheld implementation and/or coupling implementation associated withvarious types of vehicles (e.g., a land-based vehicle, a watercraft, anaircraft, a spacecraft, or other vehicle) or stationary applications(e.g., a fixed location, such as on a structure). In one embodiment,mode sensing component 160 may include communication devices that relayinformation to processing component 110 via wireless communication. Forexample, mode sensing component 160 may be adapted to receive and/orprovide information through a satellite, through a local broadcasttransmission (e.g., radio frequency), through a mobile or cellularnetwork and/or through information beacons in an infrastructure (e.g., atransportation or highway information beacon infrastructure) or variousother wired or wireless techniques (e.g., using various local area orwide area wireless standards).

In another embodiment, image capturing system 100 may include one ormore other types of sensing components 162, including environmentaland/or operational sensors, depending on the sensed application orimplementation, which provide information to processing component 110(e.g., by receiving sensor information from each sensing component 162).In various embodiments, other sensing components 162 may be adapted toprovide data and information related to environmental conditions, suchas internal and/or external temperature conditions, lighting conditions(e.g., day, night, dusk, and/or dawn), humidity levels, specific weatherconditions (e.g., sun, rain, and/or snow), distance (e.g., laserrangefinder), and/or whether a tunnel, a covered parking garage, or thatsome type of enclosure has been entered or exited. Accordingly, othersensing components 160 may include one or more conventional sensors aswould be known by those skilled in the art for monitoring variousconditions (e.g., environmental conditions) that may have an affect(e.g., on the image appearance) on the data provided by image capturecomponent 130.

In some embodiments, other sensing components 162 may include devicesthat relay information to processing component 110 via wirelesscommunication. For example, each sensing component 162 may be adapted toreceive information from a satellite, through a local broadcast (e.g.,radio frequency) transmission, through a mobile or cellular networkand/or through information beacons in an infrastructure (e.g., atransportation or highway information beacon infrastructure) or variousother wired or wireless techniques.

In various embodiments, components of image capturing system 100 may becombined and/or implemented or not, as desired or depending onapplication requirements, with image capturing system 100 representingvarious functional blocks of a system. For example, processing component110 may be combined with memory component 120, image capture component130, display component 140, and/or mode sensing component 160. Inanother example, processing component 110 may be combined with imagecapture component 130 with only certain functions of processingcomponent 110 performed by circuitry (e.g., a processor, amicroprocessor, a microcontroller, a logic device, or other circuitry)within image capture component 130. In still another example, controlcomponent 150 may be combined with one or more other components or beremotely connected to at least one other component, such as processingcomponent 110, via a wired or wireless control device so as to providecontrol signals thereto.

In one embodiment, image capturing system 100, may include acommunication component 152, such as a network interface component (NIC)adapted for communication with a network including other devices in thenetwork. In various embodiments, communication component 152 may includea wireless communication component, such as a wireless local areanetwork (WLAN) component based on the IEEE 802.11 standards, a wirelessbroadband component, mobile cellular component, a wireless satellitecomponent, or various other types of wireless communication componentsincluding radio frequency (RF), microwave frequency (MWF), and/orinfrared frequency (IRF) components adapted for communication with anetwork. As such, communication component 152 may include an antennacoupled thereto for wireless communication purposes. In otherembodiments, the communication component 152 may be adapted to interfacewith a DSL (e.g., Digital Subscriber Line) modem, a PSTN (PublicSwitched Telephone Network) modem, an Ethernet device, and/or variousother types of wired and/or wireless network communication devicesadapted for communication with a network.

In various embodiments, a network may be implemented as a single networkor a combination of multiple networks. For example, in variousembodiments, the network may include the Internet and/or one or moreintranets, landline networks, wireless networks, and/or otherappropriate types of communication networks. In another example, thenetwork may include a wireless telecommunications network (e.g.,cellular phone network) adapted to communicate with other communicationnetworks, such as the Internet. As such, in various embodiments, theimaging system 100 may be associated with a particular network link suchas for example a URL (Uniform Resource Locator), an IP (InternetProtocol) address, and/or a mobile phone number.

FIGS. 2A-B illustrate several views of an imager array 200 in accordancewith embodiments of the disclosure. Imager array 200 may be used, forexample, to implement image capture component 130 of imaging system 100.

As shown in the top view of FIG. 2A, imager array 200 may include anarray (e.g., 8 by 6 in one embodiment) of sensor arrays 202 (e.g., alsoreferred to as lenslets or optical elements). Although 48 sensor arrays202 are shown in FIG. 2A, any desired number of sensor arrays 202 may beused in other embodiments. When implemented in imager array 200, sensorarrays 202 may be of substantially equal size.

As shown in the profile view of FIG. 2B, each sensor array 202 mayinclude a plurality of sensors 206 (e.g., also referred to as pixels,elements, and sensor elements) and an associated lens 208. In oneembodiment, sensors 206 may be implemented as uncooled microbolometersensors, InGaAs sensors, or other types of sensors. In one embodiment,different sensor arrays 202 may share a common aperture through the useof a beam splitter. Sensors 206 may be provided, for example, on a base210. In one embodiment, each sensor array 202 may include an array(e.g., 80 by 80 in one embodiment) of sensors 206. Any desired number ofsensors 206 may be used in other embodiments. In one embodiment, allsensors 206 of imager array 200 may collectively provide 640 columns and480 rows of pixels. In one embodiment, imager array 200 may include oneor more read out integrated circuits (ROICs) to provide detected signalsfor processing and display.

Lenses 208 may be positioned in front of sensors 206 and separated by adistance 218. Lenses 208 may be transmissive with an appropriaterefractive index for wavebands (e.g., wavelength ranges) ofelectromagnetic radiation (e.g., irradiation) to be captured by sensors206. In one embodiment, lenses 208 may be implemented with optics ofF#1. Advantageously, by using a plurality of lenses 208 with imagerarray 200 (e.g., rather than a single lens for all of imager array 200),the focal length, associated volume of imager array 200, and optics sizemay permit a camera or other imaging system 100 to be reduced in size(e.g., by an order of magnitude in one embodiment). As a result, imagerarray 200 may be implemented as a compact, lightweight device incomparison with larger heavier conventional imagers. The small size ofimager array 200 may also permit multiple imager arrays 200 to be placedin close proximity to each other if desired.

Sensor arrays 202 may be oriented, for example, substantially in a planefacing the same direction. For distant objects (e.g., greater than 50 min one embodiment), each sensor array 202 may image the same cone inspace, and thus may capture images of the same scene 170 with negligibleparallax. Such images may be processed by appropriate components ofimaging system 100 in accordance with various techniques to provideimages of electromagnetic radiation. In one embodiment, sensor arrays202 may be placed in close proximity to each other by, for example,side-by-side placement or arranged for per-pixel filtering withassociated RGB patterns, or other patterns.

In one embodiment, a high resolution (e.g., super resolved) image may beprovided by processing images captured by multiple sensor arrays 202. Inthis regard, there may be some known phase shift (e.g., a local orglobal phase shift, by a non-integer number of pixels in someembodiments) between the various sensor arrays 202. In one embodiment,the optical resolution provided by lenses at the chosen aperture (e.g.,the diffraction limit) 208 may be higher than the sampling resolution ofsensor arrays 202.

In one embodiment, a manufacturing process for sensor arrays 202 mayresult in random pointing differences for the image cones of sensorarrays 202. In another embodiment, a high precision manufacturingprocess for sensor arrays 202 may permit exact relative pointingdifferences to be realized. In either embodiment, the final per pixelphase shift between images (e.g., also referred to as framelets)captured by sensor arrays 202 may be measured by imaging system 100using appropriate techniques.

By applying different high, low, or bandpass wavelength filters tosensor arrays 202, for example with different coating techniques and/orfilters, an effect similar to a Bayer pattern can be achieved. Themajority of sensor arrays 202 may be manufactured such that they allowtransmission of irradiance over a wide spectral band so that whencombined to a single image they achieve the highest spatial resolutionfor the most irradiance sensitive sensor arrays 202. Bandpass filteredsensor arrays 202 may also be read out at a lower frame rate allowingfor longer integration times for narrow wavebands and low irradiancepixels, thus providing high resolution and high sensitivitymultispectral imaging (e.g., for mid wave infrared imaging or otherwavebands).

Sensors 206 may be separated from each other by a plurality ofpartitions 204 provided, for example, in grid form. In one embodiment,partitions 204 may be opaque for the effective wavebands of sensors 206.As such, partitions 204 may block electromagnetic radiation outside adesired FoV of sensors 206. In this regard, as shown in FIG. 2B,electromagnetic radiation 212 and 214 passing through an angle Φ (e.g.,the half angle of the FoV) may be received by sensors 206, butelectromagnetic radiation 216 is blocked by partitions 204 and is notreceived by sensors 206. The implementation of partitions 204 in thismanner may prevent out of field objects from being imaged on adjacentsensor arrays 202. In another embodiment, custom sensors may allow forspatial separation of sensor arrays 202 such that out of fieldirradiance does not affect the neighboring sensor arrays 202.

Partitions 204 may also provide structural support for lenses 208,especially in embodiments where imager array 200 is implemented as avacuum package while lenses 208 provide the window for the vacuumpackage and receive stress associated therewith. In one embodiment, eachassociated group of sensors 206 and lens 208 in combination with itsassociated partitions 204 may effectively form a cube-like structurehaving dimensions of approximately 2 mm by 2 mm by 2 mm.

In one embodiment, imager array 200 may be implemented as a vacuumpackage with lenses 208 effectively providing both the window for thevacuum package as well as the optics for the entire imaging system 100.As a result, a complete camera or other type of imaging system may bemanufactured with fewer production steps and conventional cameras withseparate optics. Moreover, the close proximity of lenses 208 to sensors206 may permit the overall vacuum volume to be kept comparable toconventional uncooled sensors with no need for additional optics.

In various embodiments, sensor arrays 202 may perform multi spectralimaging to selectively detect desired ranges of electromagneticradiation (e.g., wavebands), such as thermal radiation, long waveinfrared (LWIR) radiation, mid wave infrared (MWIR) radiation, shortwave infrared (SWIR) radiation, near infrared (NIR) radiation, visiblelight (VIS), and/or other ranges. In this regard, lenses 208 may includeappropriate coatings, or sensor arrays 202 may be provided withappropriate filters, to filter the electromagnetic radiation received bysensors 206. As a result, different sensor arrays 202 may detectdifferent broad or narrow bands of electromagnetic radiation. In oneembodiment, at least five spectral bands may be detected (e.g., rangingfrom visible light to LWIR, or other ranges).

For example, in one embodiment, a group 220 of sensor arrays 202 mayinclude filters to detect red visible light, a group 222 of sensorarrays 202 may include filters to detect green visible light, a group224 of sensor arrays 202 may include filters to detect blue visiblelight (e.g., groups 220, 222, and 224 may provide RGB patterns), and agroup 226 of sensor arrays 202 may include filters to detect NIR/S WIRradiation (e.g., approximately 700-1700 nm). Other configurations,groupings, and detection ranges may be provided in other embodiments.For example, different sensor arrays 202 may use different types ofsensors 206 to detect different wavebands (e.g., InGAs sensors may beused to detect VIS-SWIR wavebands, and bolometer sensors may be used todetect MWIR-LWIR wavebands).

Multi spectral imaging may have dramatic advantages over single wavebandimaging, and may be used in a variety of different applications such asgeo sensing, target detection, target, classification, and targettracking using multiple sensor arrays 202 for improved capabilities andperformance. By processing images from different combinations ofwavebands and different phase shifted sensor arrays 202, images may becreated that provide reasonable spatial resolution, excellent low lightperformance, and multi-spectral information about scene 170.

Wavebands from NIR to LWIR may show very different properties and aremore or less suited for a specific imaging applications under specificenvironmental conditions. Factors such as vapor content in atmosphere,particle sizes in dust or aerosols, and scene dynamic range might rendera MWIR sensor array useless but have no or only very limited effect on aNIR or LWIR sensor array. In addition, specific materials may havespecific spectral signatures. By capturing a scene using multiplewavebands, the response profile can be compared to a database ofnormalized known spectral responses. As a result, imaging system 100 mayattempt to classify the material.

Table 1 identifies various parameters of imaging array 200 in anembodiment configured to operate in the LWIR waveband.

TABLE 1 Property Value Imaging array 48 sensor arrays arranged in 8 by 6matrix Imaging array size 16 mm by 12 mm Sensor array 6400 sensors in 80by 80 matrix Sensor array size 2 mm by 2 mm Sensor pitch 25 μm Focallength 2.5 mm F-number 1.25 Normalized wave length 10 μm Effective FoV44° Airy disc diameter 1.22 pixels (first minima)

Table 2 identifies various parameters of imaging array 200 in anembodiment configured to operate in the VIS-NIR waveband (e.g., usingInGaAs sensors capable of performing extended detection into the visiblewaveband down to, for example, 350 nm).

TABLE 2 Property Value Imaging array 48 sensor arrays arranged in 8 by 6matrix Imaging array size 16 mm by 12 mm Sensor array 6400 sensors in 80by 80 matrix Sensor array size 2 mm by 2 mm Sensor pitch 25 μm Focallength 4 mm F-number 2 Normalized wave length 1300 nm Effective FoV 28°Airy disc diameter 0.25 pixels (first minima)

In one embodiment, sensor arrays 202 may exhibit reduced size incomparison to many existing imaging devices. For example, the use offilters or lens coatings at each sensor array 202 may permit desiredwavebands spectra to be detected without requiring the use of largeexternal optics or filter wheels, thus reducing size.

In one embodiment, individual sensor arrays 202 may be smaller thanconventional image sensors. For example, in one example, an array ofsensor arrays 202 may exhibit approximately the same surface area as asingle conventional sensor array. By providing a lens 208 in each sensorarray 202, such a configuration need not be diffraction limited in themanner of conventional high resolution sensors (e.g., greater than 640by 480 pixel resolution) having small pitch sensor elements (e.g., lessthan 20 microns) where the spatial resolution of the optics may set theabsolute diffraction limit for conventional sensors. In one embodiment,the diffraction limit may be set by the size of aperture.

The various features of imager array 200 may be used in a variety ofapplications to great advantage. For example, in one embodiment, imagerarray 200 may be modified to support foveal imaging.

In this regard, FIGS. 2C-D illustrate several views of an imager array230 in accordance with embodiments of the disclosure. Imager array 230may be used, for example, to implement image capture component 130 ofimaging system 100. Imager array 230 includes a plurality of sensorarrays 202 and 232 of different sizes and having different focallengths. In this regard, in the higher frequency wavebands (e.g.,VIS-SWIR), longer focal lengths may be used without the risk of beingdistortion limited by optics.

As shown in the top view of FIG. 2C, imager array 230 may include anarray (e.g., 4 by 6 in one embodiment) of sensor arrays 202. Although 24sensor arrays 202 are shown in FIG. 2C, any desired number of sensorarrays 202 may be used in other embodiments.

Imager array 230 may also include an array (e.g., 2 by 3 in oneembodiment) of sensor arrays 232. Although 6 sensor arrays 232 are shownin FIG. 2C, any desired number of sensor arrays 232 may be used in otherembodiments.

As shown in the profile view of FIG. 2D, each sensor array 232 mayinclude a plurality of sensors 236 (e.g., also referred to as pixels)and an associated lens 238. In one embodiment, each sensor array 232 mayinclude an array of sensors 236. Any desired number of sensors 236 maybe used in various embodiments. Lenses 238 may be positioned in front ofsensors 236 and separated by a distance 248.

As shown in FIG. 2D, distance 248 may be greater than distance 218. Inthis regard, sensor arrays 232 may exhibit a greater focal length thansensor arrays 202.

Sensor arrays 202 and 232 may be oriented, for example, substantially ina plane facing the same direction and configured to detect images fromscene 170. Such images may be processed by appropriate components ofimaging system 100 in accordance with various techniques to provideimages of electromagnetic radiation.

Sensors 232 may be separated from each other by a plurality ofpartitions 234 provided, for example, in grid form. In one embodiment,partitions 234 may block electromagnetic radiation outside a desired FoVof sensors 234. In this regard, as shown in FIG. 2D, electromagneticradiation 242 and 244 passing through an angle ρ (e.g., the half angleof the FoV) may be received by sensors 236, but electromagneticradiation outside the FoV is blocked by partitions 234 and is notreceived by sensors 236.

In various embodiments, sensor arrays 202 and 232 may detect the same ordifferent ranges of electromagnetic radiation. In this regard, lenses208 and 238 may include the same or different coatings, or sensor arrays202 and 232 may be provided with the same or different filters, tofilter the electromagnetic radiation received by sensors 206 and 236.

As shown in FIG. 2D, sensor arrays 202 may exhibit a wider FoV (e.g.,twice as large in one embodiment) than sensor arrays 232. Also, sensorarrays 232 may include a larger number of sensors 236 (e.g., four timesas many in one embodiment) than the sensors 206 of sensor arrays 202.

As a result, sensor arrays 202 may capture images having a relativelywide FoV and relatively low resolution (e.g., to capture a low spatialfrequency image). Such low resolution images may be provided, forexample, to a remote observer as video images over a low bandwidthconnection that may not be able to support the bandwidth associated withvery high resolution images. In comparison, sensor arrays 232 maycapture images having a relatively narrow FoV and relatively highresolution (e.g., to capture a high spatial frequency image). In someembodiments, such configurations permit sensor arrays 232 to receivemore irradiance samples for a given FoV than sensor arrays 202. In thisregard, sensor arrays 202 and 232 may be used to provide foveal imaging(e.g., to permit a human or machine observer to monitor a wide FoV imageof scene 170 and also view a detailed, higher spatial resolution, narrowFoV image within scene 170).

For example, sensor arrays 202 and 232 may be implemented such thattheir optical centers approximately match each other. Thus, the narrowFoV images provided by sensor arrays 232 may provide a high spatialresolution sampling in the center of wide FoV images provided by sensorarrays 202 having a lower spatial resolution. Such an embodiment maypermit foveal imaging in which a wide FoV image is captured (e.g., usingsensor arrays 202) while a narrow Fov image is also captured (e.g.,using sensor arrays 232) to permit fine spatial details of scene 170 tobe resolved if desired.

In one embodiment, multiple sensor arrays 232 may be combined and mappedto provide an even higher spatial resolution grid in the center of theFoV. For example, four sensor arrays 232 may be combined into a singlenarrow FoV image with a sampling ratio four times higher than one ofsensor arrays 232 alone.

Advantageously, the use of imaging array 230 to perform foveal imagingmay avoid various limitations associated with conventional fovealimaging techniques including, for example, large aperture optics, highcost of manufacture, complexity (e.g., of multi FoV optics), parallax,or other limitations.

Although various references are made to imager array 200 and sensorarrays 202 in this disclosure with regard to various features, suchfeatures may be similarly provided by imager array 230 and sensor arrays232 where appropriate.

In one embodiment, to minimize size, complexity, power consumption, andcost, sensors 206 may be implemented as uncooled microbolometer sensorsfor the LWIR waveband. Highpass filtering may be applied to signalsprovided by such sensors 206 to permit detection in the MWIR wavebandfor stable targets in scene 170 if used with lenses 208. In oneembodiment, a large aperture (e.g., a low F#) may be used to receivesufficient MWIR radiation to perform imaging.

When sensors 206 are implemented as LWIR sensors, the optical design andconfiguration of sensor arrays 202 may be diffraction limited. Forexample, FIG. 2E identifies diffraction limits for different fields ofview in accordance with an embodiment of the disclosure. In FIG. 2E,sensor array 202 is implemented with an array of 80 by 80 sensors 206with 25μ pitch as suggested by the diameter of the airy disc firstminima. As shown, approximately 50 percent of the energy is containedwithin a circle half the size of the airy disc. In one embodiment,larger sensors 206 (e.g., pixels) may be desirable for improvedsensitivity and may permit sensor arrays 202 exhibit optical resolutionhigher than sensor resolution (e.g., individual sensors 206 mayundersample scene 170).

FIG. 2F identifies the size of an airy disc as a function of wavelengthin accordance with an embodiment of the disclosure. In FIG. 2F, the sizeof the airy disc increases linearly with wave length (e.g., for opticswith F# approximately equal to 1 and sensors 206 with 25μ pitch).

In another embodiment, imager array 200 may be used to provide imageswith a high signal to noise ratio (SNR). In this regard, conventionalthermal imagers (e.g., particularly uncooled systems usingmicrobolometers) often suffer from high spatial and temporal noise. Suchnoise may limit an imager's ability to detect small changes inirradiation.

Unfortunately, many conventional approaches to increasing the signal tonoise ratio are impractical or overly complex. For example, oneparameter that directly affects SNR is the amount of irradiance (e.g.,the power of electromagnetic radiation per unit area at a surface) thatcan be absorbed per time unit. In a conventional microbolometer basedimager, the surface area of the sensors may at least partly determinehow much irradiance may be collected. However, increasing the size ofindividual sensors may result in fewer sensors being provided in thesame size imager. Such an approach may result in drastically reducedspatial resolution (e.g., due to fewer sensors), reduced reliability(e.g., due to fewer sensors remaining in case of sensor failure), andhigher costs (e.g., due to larger optics and the complexity of specialsensor geometries).

As another example, larger aperture optics may be used to collect moreirradiance per time unit to improve the SNR. However, such an approachmay require larger optical elements that add weight and bulk materialcost, and may require complicated manufacturing techniques.

As a further example, higher gain signal amplifiers may be used in theanalog domain to improve the SNR. However, high gain may be difficult toachieve while still maintaining linearity. In addition, a high gainstage may limit the dynamic range of an imaging system because the limitof the analog to digital (A/D) converters may be reached at lowerirradiance levels.

As yet another example, post processing of image data (e.g., thedigitized signal) may improve the SNR. However, such processing mayintroduce unwanted artifacts such as blur, and may not always be able toseparate noise from actual scene irradiance.

In contrast to such approaches, imager array 200 may provide improvedSNR through the use of multiple sensor arrays 202 imaging approximatelythe same scene 170. In this regard, signals from multiple sensor arrays202 may be combined to provide a virtual sensor image with a higher SNRthan exhibited by the images provided by individual sensor arrays 202.

For example, FIG. 3 illustrates a process of obtaining an image having adesirable signal to noise ratio in accordance with an embodiment of thedisclosure. In block 302, sensors 206 of multiple sensor arrays 202 maybe mapped to a virtual sensor grid (e.g., a set of pixels). Sensorarrays 202 may capture images of scene 170 (block 304).

By lowering the resolution of the virtual sensor grid, an improved SNRmay be achieved that is approximately proportional to the amount ofcumulative signal mapped to each location in the virtual sensor grid.For example, if the resolution of the virtual sensor grid is one quarter(¼) of the resolution of the entire imager array 200 in both thevertical and horizontal dimensions (e.g., the number of all sensors 206in all sensor arrays 202 combined), then each pixel of the virtualsensor grid may accumulate signals from multiple sensor elements (block306). For example, in one embodiment, each virtual sensor grid mayaccumulate signals from 16 sensor arrays 202). The resulting image(e.g., a result image) associated with the virtual sensor grid mayexhibit a higher SNR than images from individual sensor arrays 202(block 308). In this regard, if random noise has a zero mean, then thenoise of the virtual sensor grid (e.g., having a lower resolution) maybe one quarter of that of the actual signals from sensor arrays 202(e.g., noise may be reduced as the square root of the number ofsamples).

By lowering the spatial and temporal noise in accordance with theprocess of FIG. 3, the detection range of imager array 200 may beimproved. Such improvement may be particularly useful, for example, forsurveillance cameras used in applications such as perimeter protection.

In another embodiment, imager array 200 may be used to provide highresolution images by taking advantage of predetermined phase shiftsbetween different sensor arrays 202. For example, FIG. 4 illustrates aprocess of obtaining a high resolution image using phase shifts betweensensor arrays 202 in accordance with an embodiment of the disclosure. Inone embodiment, the process of FIG. 4 may be implemented using superresolution processing techniques.

Intentional or unintentional production variations of the relativepositions of sensors 206 and/or lenses 208 may cause different sensorarrays 202 to capture images from slightly different scenes 170 (e.g.,non-identical locations resulting in phase shifts between images fromdifferent sensor arrays 202). Super resolution processing techniques maybe used to combine phase shifted images from the different sensor arrays202 into a single, super resolved, image. For example, in oneembodiment, such super resolution processing may be used to combine andconvert low resolution images of approximately 80 by 80 pixels up tohigh resolution images of approximately 320 by 240 pixels or close tothe diffraction limit of the optics.

For example, in block 402 of FIG. 4, lenses 208 of different sensorarrays 202 may be slightly shifted relative to each other such that thecenter of the optical axis for each sensor array 202 slightly differsfrom other sensor arrays 202. In one embodiment, these differences inthe optical axis (e.g., horizontal and vertical offsets) may becalibrated, measured, and determined for each sensor array 202 relativeto a reference sensor array 200. Such operations may be performed, forexample, at the time sensor arrays 202 are manufactured (e.g., thuseliminating the need for complex and error prone real time optical flowcalculations). As a result, although sensor arrays 202 may be positionedto face the same scene 170, the electromagnetic radiation received byeach sensor array 202 may be phase shifted (e.g., exhibiting a sub pixelphase shift) by a known amount relative to that received by other sensorarrays 202 (block 404). As a result, images captured by each sensorarray 202 may be phase shifted relative to the images captured by othersensor arrays 202 (block 406).

Thus, by varying the alignment of the center of the optical axis foreach sensor array 202, captured images may exhibit arbitrary sub pixelphase shifts. For distant scenes 170, parallax effects associated withthe spatial separation in the image plane may be negligible.

The phase shifted images captured by multiple sensor arrays 202 may becombined and/or otherwise processed to provide a higher resolution image(e.g., a result image) than would otherwise be provided by the imagescaptured by individual sensor arrays 202 (block 408).

Advantageously, by combining phase shifted images (e.g., on a per pixellevel in one embodiment), a higher scene sampling rate may be achieved.In one embodiment, the optical resolution provided by lenses 208 may behigher than the sampling resolution of sensors 206. In this case, bycombining phase shifted images from multiple sensor arrays 202 andapplying an appropriate Wiener filter or other deconvolution method, theresulting image may exhibit a higher resolution (e.g., approximately twoto three times higher in one embodiment) than that of images provided byindividual sensor arrays 202. In one embodiment, the process of FIG. 4may be performed automatically.

In one embodiment, the processes of FIGS. 3 and 4 may be combined topermit imaging system 100 to run in several different modes. Forexample, in one mode, a low resolution image with a low SNR may beprovided in accordance with the process of FIG. 3. In another mode, ahigher resolution image may be provided in accordance with the processof FIG. 4. In yet another mode, the processes of FIGS. 3 and 4 may beperformed simultaneously (e.g., to provide different result images usingdifferent processes). Other processes provided in this disclosure may becombined where appropriate as may be desired in particular applications.

In another embodiment, imager array 200 may be used to provide stereoimaging (e.g., stereo vision). For example, FIG. 5 illustrates imagerarray 200 configured to provide stereo imaging in accordance with anembodiment of the disclosure.

As discussed, in certain embodiments (e.g., for objects at a distancegreater than 50 m in one embodiment), parallax caused by the relativespacing between sensor arrays 202 may be negligible. However, in otherembodiments, such parallax may be used to provide stereo (e.g., threedimensional and/or depth imaging from a single pair of images capturedby any two sensor arrays 202) images of scene 170.

For example, in FIG. 5, an object X may be positioned in scene 170 at ashort distance A (e.g., less than approximately 5 m in one embodiment)from imager array 200. Object X may be shifted relative to the opticalaxis of a lens 208B by a distance B, and shifted relative to the opticalaxis of a lens 208C by a distance C. In the embodiment shown in FIG. 5,lenses 208B and 208C may have a focal length of approximately 2 mm.

As shown in FIG. 5, electromagnetic radiation from object X may bereceived by lens 208B at an angle Ø1 relative to the optical axis oflens 208B, and received by lens 208C at a different angle Ø2 relative tothe optical axis of lens 208C. As a result, when sensors 206B associatedwith lens 208B capture an image of scene 170, object X may be offsetfrom the center of the image by a distance D1. However, when sensors206C associated with lens 208C capture an image of scene 170, object Xmay be offset from the center of the image by a different distance D2.

In one embodiment, the different images provided by sensors 206B and206C may be used to provide stereo vision, for example, in the form ofrealtime stereo video images or static stereo images. Such imagesprovide a user with a three dimensional view of object X in scene 170.

Such stereo images may be used in a variety of applications. Forexample, imager array 200 may be provided in a thermal imaging cube foruse in hazardous environments, such as by firefighters or otheremergency personnel to provide three dimensional images of a hazardousenvironment. Such images may be transmitted wirelessly or by wire fromthe hazardous environment to safe locations for viewing.

In another embodiment, a plurality of sensor arrays 202 may beconfigured to detect images from electromagnetic radiation receivedthrough a plurality of apertures distributed over the outside surface ofa device to provide a robust detector that may be thrown or otherwiseintroduced to a hazardous environment, such a smoke filled space. Such adevice may be configured to wirelessly transmit images (e.g., infrared,multi-spectral, or other images) to a non-hazardous location to permitusers to safely view the hazardous environment (e.g., in a 360 degreefield of view).

In another embodiment, imager array 200 may be used to provide redundantsensor arrays 202 that permit imager array 200 to provide high qualityimages despite the presence of possible defects in one or more sensors206. In this regard, modern high spatial resolution imaging devices areexpensive, complex devices and may be subject to stringent manufacturingtolerances. Indeed, for many imaging devices, the imager (e.g.,detector) may be the single most expensive component. Microbolometerthermal imaging micro-electromechanical systems (MEMS) devices withsmall dimensions (e.g., small pixel pitch) may have productionparameters that are particularly difficult to meet consistently. Suchproduction parameters may include, for example, clean roomspecifications, production equipment, process repeatability, rawmaterial purity, manual handling of the completed parts, and otherparameters. Variations on any of the production parameters may lead todecreased yields (e.g., due to defective devices) which increase theoverall cost for each specification-compliant device.

For thermal imaging devices in particular, imperfections in productionmay result in any number of non operating sensors. For high resolutiondevices, for example devices with 640 by 480 sensors or more, it may bedifficult to produce devices with 100 percent operability (e.g., whereinevery pixel operates within specification under all operatingconditions).

As a result, producers (e.g., manufacturers) of imaging devices mayspecify some maximum number of non operating pixels. For example,producers may set the permissible number of defective pixels to 0.1percent of all pixels, or may limit the number of defective pixels inthe center of the images to a small number, but permit larger numbers ofdefective pixels to be present in peripheral parts of the images. Asanother example, producers may limit the number of permissible seriousdefects, such as entirely defective rows or columns. In particular, itmay be difficult to replace values from two or more neighboring rows orcolumns. It is therefore typical for producers to reject or discarddevices that include adjacent defective rows or columns or clusters ofdefective pixels.

Conventional corrective techniques are often ineffective for largeclusters of defective pixels. Moreover, it is often impractical to reusedevices with defective pixels in other lower resolution products.

FIG. 6A illustrates a process of correcting defective pixels (e.g.,sensors 206) in imager array 200 in accordance with an embodiment of thedisclosure. One or more defective sensors 206 may be detected (block602), and the corresponding sensor arrays 202 of imager array 200including the defective sensors 206 may be identified (block 604). Invarious embodiments, such detection and identification may be performedduring the manufacture and testing of imager array 200, or during thesubsequent operation of imager array 200 in the field.

For example, FIGS. 6B and 6C illustrate sets 620 and 630 of variousimages 622 and 632, respectively, captured by 12 sensor arrays 202(e.g., a subset of sensor arrays 202). As shown in FIG. 6B, one ofimages 622 includes a cluster of defective pixels 624 which largelyobscure the information shown by the defective image. In this regard, acluster of sensors 206 in one of sensor arrays 202 are defective andfail to provide useable image data of the captured scene. As also shownin FIG. 6B, the remaining eleven images 622 do not include defectivepixels and are provided by sensor arrays 202 with working sensors 206.

In FIG. 6C, three of images 632 include various clusters of defectivepixels 634 which span multiple rows and columns, and largely obscure theinformation shown by the three defective images. As also shown in FIG.6C, the remaining nine images 632 do not include defective pixels andare provided by sensor arrays 202 with working sensors 206.

Advantageously, sensor arrays 202 may capture at least partiallyredundant images of the same scene 170. As a result, imaging system 100may disregard the defective images provided by sensor arrays 202 withdefective sensors 206, or correct the defective images or pixels withappropriate image data from working sensors 206 in other sensor arrays202 (block 606). As a result, imaging system 100 may provide a correctedimage (e.g., a result image) that includes all defective pixels filledin with appropriate image data (block 608).

Moreover, in embodiments where different sensor arrays 202 exhibitslightly different optical alignment and local distortion, imagesprovided by different sensor arrays 202 may not be entirely identical.Such differences may permit interpolation techniques to be used tocorrect the defective image data.

In another embodiment, imager array 200 may be calibrated without theuse of a shutter. Infrared cameras in the MWIR and LWIR wavebands aresensitive to thermal radiation. Unlike a visible spectrum camera thatmay be built such that visible light may only enter through the optics,a thermal camera may generate infrared radiation from sources inside thethermal camera. For example, electronics may generate significantamounts of infrared radiation (e.g., irradiance). Unfortunately, thesesources of irradiation that are not from the scene to be imaged maynevertheless be measured by infrared camera sensors (e.g., infraredradiation from a heat source inside the thermal camera may reflect offsurfaces inside the imager and end up detected by the infrared camerasensors.

One conventional approach to compensate for such internal infraredradiation in cooled and uncooled thermal imagers is to perform flatfield correction (FFC). In this regard, detection of the scene may betemporarily blocked by inserting an opaque object (e.g., a shutter) inthe optical path (e.g., assuming that signals measured by the sensorswhile the optical path is blocked stay constant or nearly constant). Bymeasuring signals detected by the sensors while the optical path isblocked (e.g., fixed pattern noise (FPN)), and subtracting such signalsfrom signals detected while the optical path is not blocked, images maybe provided that include only scene information.

Unfortunately, such a conventional approach typically involves the useof a moving shutter which may add complexity and cost to an imagingsystem, and may compromise reliability. Moreover, calibration performedwith a shutter may temporarily render an imaging system blind to thescene. In addition, a single shutter with a constant, uniformtemperature does not allow for gain calibration (e.g., offset correctiononly) which may result in image artifacts, especially for high dynamicrange scenes.

Another approach to compensate for such internal infrared radiation isto perform signal processing, such as scene based non uniformitycorrection (SBNUC) processing that relies on comparisons between two ormore video frames. If there is some frame to frame motion, either due tothe imager moving relative to the scene or some object in the scenemoving, the measured irradiance at one sensor element location may becompared to another sensor element location in another video frame.Under the assumption that the scene irradiance stays constant, it isexpected that all sensor elements should measure the same irradiancelevel for a given point in the scene. If different levels of irradianceare measured, this may be interpreted to be the result of out of fieldirradiance (e.g., FPN) corrupting the image.

Unfortunately, such SBNUC approaches generally require some frame toframe motion that is known to some degree of accuracy. For example,motion may be image based (e.g., calculated based on the sceneinformation) or non image based (e.g., calculated based on an externalmotion sensor such as a MEMS gyroscope). Unfortunately, image basedmotion estimation approaches tend to fail when the scene dynamic rangeis small and/or the amount of FPN is large (e.g., where the SNR ispoor). Non image based motion estimation approaches tend to fail whenthere is scene deformation or intra scene motion (e.g., a person or carmoving relative to the scene).

Imager array 200 may be calibrated using several alternatives to theabove approaches. For example, sensor arrays 202 may capture multipleimages of the same scene 170 simultaneously, or substantiallysimultaneously depending on the exact properties of sensors 206 andtheir associated ROICs.

For example, in an embodiment with 48 sensor arrays 202, 48 images ofscene 170 may be captured substantially simultaneously. During opticscharacterization, it can be exactly determined which sensors 206 in eachsensor array 202 correspond to sensors 206 in other sensor arrays 202.The mean or median value of independent signals (e.g., data) detected bythe corresponding sensors 206 (e.g., corresponding to a single point inthe scene) may be used to correct all the corresponding sensors 206.Such an approach may be used in arbitrarily poor SNR conditions, doesnot require imager or scene motion, does not require moving parts, andis immune to frame to frame scene deformation. Accordingly, such anapproach has clear benefits to conventional approaches for reducing FPN.

FIG. 7A illustrates a process of calibrating sensors 206 of imager array200 in accordance with an embodiment of the disclosure. Advantageously,the process of FIG. 7A may be performed without a moving shutter andwithout obscuring scene 170 from view by imager array 200. In oneembodiment, depending on the repeatability and accuracy of the design,manufacture, and assembly of sensor arrays 202, it may be possible todetermine which sensor 206 in one sensor array 202 corresponds to othersensors 206 in other sensor arrays 202 (e.g., corresponding to the samecoordinate or pixel of scene 170). However, if some production variationexists, then each sensor array 202 may be tested to determine suchcorrespondence.

In this regard, sensor arrays 202 may be used to image a target (e.g.,scene 170) having a known pattern and placed sufficiently far enoughaway to render any parallax negligible. For example, a collimator may beused to produce a target at infinity.

Individual sensor array 202 distortion coefficients 702 may identifyoffsets between individual sensors 206 and a global scene coordinatespace. In one embodiment, the global scene coordinate space may bedivided into discrete scene coordinates (e.g., scene pixels) at aresolution identical to that of individual sensor arrays 202. In oneembodiment, the global scene coordinate space may correspond to a mastersensor array 202.

Thus, distortion coefficients may be expressed relative to an idealscene mapping provided by the global scene coordinate space. Forexample, the distortion coefficients may be expressed as vertical andhorizontal offset values relative to the global scene coordinate space.Distortion coefficients 702 may be stored, for example, in a nonvolatile memory provided on imager array 200 or imaging system 100.

If imager array 200 is intended to be used at sufficiently small objectdistances such that parallax effects may render distortion coefficients702 invalid, then uncorrected data 701 or distortion coefficients 702may be optionally offset by appropriate parallax compensation values inaccordance with a parallax compensation process in block 705. In thisregard, because the distance between the optical centers of each sensorarray 202 may be known, parallax effects may be readily determined inaccordance with conventional techniques. Because parallax effects arestronger for closer objects, the parallax compensation process in block705 may receive measurements or estimates of the distance between imagerarray 200 and scene 170 from a parallax estimation process in block 703,or from a distance sensor 704 (e.g., a laser range finder).

Alternatively, the parallax estimation process in block 703 may analyzeimages captured by each sensor array 202 and match common features. Forexample, conventional corner detection feature extraction processes maybe used. As another example, block matching may be used to measure theamount of parallax. If the focal lengths of lenses 208 are known, andthe distance between their optical centers is known, then the distancefrom each sensor array 202 to scene 170 becomes proportional to theparallax.

If multiple features (e.g., corners) are matched, multiple localparallax estimates may be calculated. These estimates may be averaged toprovide a more accurate average scene distance, or they may be locallyinterpolated to provide a local distance map with spatially varyingparallax. Observed object shifts due to parallax may be used to modifythe distortion coefficients in the parallax compensation process ofblock 705.

In one embodiment, the parallax compensation process of block 705 may beperformed (e.g., using appropriate processing or optics) by mappingimages from sensor arrays 202 to locations on a super resolved grid. Forexample, the amount of parallax associated with a given sensor array 202may be estimated by measuring the positions of visible non-occludedobjects in scene 170 in all sensor arrays 202, or by using anappropriate external distance measuring device.

In block 706, uncorrected data 701 (e.g., signals, samples, or datavalues, such as pixel values) captured by each sensor 206 of sensorarrays 202 may be mapped, for example using a forward transform, to theglobal scene coordinate space by applying distortion coefficients 702(e.g., optionally further offset for parallax compensation). In oneembodiment, each sensor 206 (e.g., and its corresponding pixel) of eachsensor array 202 may be mapped to a corresponding coordinate of thescene coordinate space, for example, by selecting a scene coordinatehaving a center that closest matches the center of the correspondingsensor 206. Appropriate interpolation techniques (e.g., using nearestneighbor approximations or other techniques) may also be used to mapeach sensor 206 to a corresponding scene coordinate (e.g., if a preciseinteger match is not available).

For each scene coordinate, sampled values (e.g., detected data) may beaccumulated from one corresponding sensor 206 of each sensor array 202to provide an accumulated value for each scene coordinate (block 707).For example, in one embodiment, 48 sensor arrays 202 may be provided.Accordingly, 48 sampled values (e.g., each value being provided by acorresponding sensor 206 in each of sensor arrays 202) may beaccumulated for each scene coordinate.

In one embodiment, some sensors 206 of some sensor arrays 202 may not bemapped to scene coordinates because local distortion may be such that aparticular sensor 206 images a location that is not part of a common FoVfor the collective set of sensor arrays 202.

In one embodiment, a reference (e.g., “correct”) scene irradiance (e.g.,data value) may be determined for each scene coordinate and may be themean of the sampled values (e.g., sampled irradiance levels) detected bythe corresponding sensors 206 of sensor arrays 202. For example, themean may be calculated for a scene coordinate by dividing theaccumulated value for the scene coordinate by the number of sensors 206mapped to the scene coordinate. To prevent overwriting the accumulatedvalues when sampled values for the next image frame is available, twoaccumulators may be used in block 707, with one accumulator beingwritten while the other accumulator is used to calculate offsetcorrection terms (e.g., values) in block 708 as will be described.

In block 708, offset correction terms (e.g., values) may be calculatedfor all sensors 206. For example, if there are M×N sensor arrays 202,and each sensor array 202 has R×C sensors 206, then there may be a totalof M×N×R×C offset correction terms.

In one embodiment, the offset correction term for a particular sensor206 may be calculated by taking the difference between: the mean of thesampled values for the scene coordinate corresponding to the particularsensor 206; and the actual sampled value detected by the particularsensor 206. The offset correction terms may be stored in an offsetcorrection term map (block 714).

Also in block 708, a set of gain correction terms may be determined. Forexample, one set (e.g., an image frame) of uncorrected data valuescaptured at a time T0 may be stored and compared to another set capturedat a time T1.

For any sensor 206, if the difference in the data values captured attimes T0 and T1 is significantly larger than the expected noise, then itmay be determined that the irradiance has increased or decreased. Thisdifference may be independent of any offset error.

By comparing these differences for all sensors 206 measuring irradiancefrom the same location in scene 170 (e.g., all sensors corresponding tothe same scene coordinate), a gain term may be determined for eachsensor 206 to cause the relative responsivity of sensors 206 to benormalized (e.g., made equal) to each other (e.g., assuming that the FPNhas not changed significantly between times T0 and T1). The gain termsmay be stored in block 712.

For example, a mean V0 of sampled sensor values for a scene coordinateat time T0 may be calculated and stored in a memory. At later time T1, amean V1 may be calculated and stored for the same scene coordinateexhibiting a change in irradiance. The scene coordinate may be mapped toa corresponding sensor 206 of each sensor array 202 (e.g., using inversedistortion coefficients further described herein). If a precise mappingis not available, a nearest neighbor sensor 206 may be chosen, orappropriate interpolation techniques may be used.

A difference D between the mean values (D=V1−V0) may represent the meanresponse to the change in irradiance in scene 170. If v0 and v1represent the irradiance measured by a particular sensor 206 in aparticular sensor array 202, then a difference d may represent theresponse of the particular sensor 206 (d=v0−v1) to the change inirradiance in scene 170. Accordingly, the gain correction term for theparticular sensor may be Did. For example, if D=10 and d=20, then theindividual sensor 206 may be twice as responsive as the mean of allcorresponding sensors 206, and the gain of the individual sensor 206 maytherefore be adjusted by a gain term of 0.5 to normalize its response.

In one embodiment, the process of FIG. 7A may be performed iterativelysuch that offset correction terms 714 may be repeatedly updated. In thisregard, an optional damping process (block 713) may be used to damp therate of change of the offset correction terms by calculating a dampedoffset term using a weighted average of a previously stored offsetcorrection term and a newly calculated offset correction term. In thisregard, offset correction terms and gain correction terms may be dampedusing the process of block 713, thus reducing the effects of dramaticsample value differences in scene 170 with very strong gradients due to,for example, imperfectly modeled distortion effects on a coarse-grainedsensor array 202.

As shown in FIG. 7A, gain terms 712 may be applied to uncorrected data701 in block 709. Offset correction terms 714 may be applied to thegain-adjusted uncorrected data 701 (block 710) to provide corrected data711. Corrected data 711 may also be used by the parallax estimationprocess in block 703.

In one embodiment, the process of FIG. 7A may be a nearest neighborapproximation of a more general process identified in FIG. 7B. In thisregard, FIG. 7B illustrates another process of calibrating sensors 206of imager array 200 in accordance with an embodiment of the disclosure.Similar to FIG. 7A, the process of FIG. 7B may also be performed withouta moving shutter and without obscuring scene 170 from view of imagerarray 200.

In the process of FIG. 7B, data values detected by sensors 206 of eachsensor array 202 may be compared to one or more data values detected bysensors 206 of other sensor arrays 202 corresponding to the samelocation in scene 170. Differences between the data values may beaccumulated to provide offset correction terms. In one embodiment, thedata values of the sensors 206 of other sensor arrays 202 may bedetermined by performing an interpolation between some number of closestneighbor sensors 206.

Uncorrected data 701 may be provided to a frame buffer (block 721) andpassed to block 706 where uncorrected data 701 may be mapped usingdistortion coefficients 702 in the manner previously described withregard to FIG. 7A. As such, each sensor 206 of each sensor array 202 maybe mapped to a corresponding scene coordinate using distortioncoefficients 702.

In addition, each scene coordinate may be mapped (e.g., also referred toas a reverse transform) to a corresponding sensor 206 in each of thesensor arrays 202 using inverse distortion coefficients 727 (e.g., alsoreferred to as reverse distortion coefficients). For example, in oneembodiment, each scene coordinate may be mapped to 48 differentindividual sensors 206 in 48 respective sensor arrays 202. Therefore, inthis embodiment, 48 sets of inverse distortion coefficients 727 may beprovided for each scene coordinate (e.g., with each set including ahorizontal coefficient and a vertical coefficient) to map each scenecoordinate to corresponding sensors 206 (block 726).

Appropriate interpolation techniques (e.g., using a linear combinationof multiple nearest neighbors or other techniques) may also be used tomap a scene coordinate to a corresponding sensor 206 in each sensorarray 202 (e.g., if a precise integer match is not available) and todetermine the sample value associated with the corresponding sensor 206.For example, each sensor 206 of a given sensor array 202 may be mappedto locations (e.g., corresponding sensors 206) of other sensor arrays202 by, for example, bilinear interpolation of the four nearest inversedistortion coefficients 727.

Because the distortion (e.g., offsets) between sensors 206 relative toother sensors 206 of other sensor arrays 202 is constant, the mappingfrom a first sensor 206 in a first sensor array 202 to othercorresponding sensors 206 in other sensor arrays 202 (e.g., usingdistortion coefficients 702 to map the first sensor 206 to a scenecoordinate, and using inverse distortion coefficients 727 to map thescene coordinate the corresponding sensors 206 of other sensor arrays202) may be pre calculated and stored, for example, in a table for eachsensor 206. Thus, using the mapping determined in block 726, each sensor206 of each sensor array 202 may be mapped to corresponding sensors 206of the other sensor arrays 202.

In block 734, the sampled value of each sensor 206 may be compared withthe sampled values of all other mapped corresponding sensors 206.Differences between the sampled value of each sensor 206 and the sampledvalues of its corresponding mapped sensors 206 may be accumulated (block734). The accumulated differences may be used to calculate offsetcorrection terms in block 708 in the manner described in FIG. 7A.

As shown in FIG. 7B, additional processing may be performed in blocks708, 709, 710, 712, 713, and 714 as described in FIG. 7A to providecorrected data 732. In various embodiments, the processing of FIGS. 7Aand 7B may be performed automatically.

In another embodiment, imager array 200 may be used to perform gasdetection. Many gases are transparent in visible light. Some of thesetransparent gases may be directly harmful to humans or may have short orlong term negative impacts on the environment. It is therefore importantto detect emissions of such gases.

However, conventional multispectral systems used for gas detection areoften complex and expensive. They often require multiple detectorsarranged in complex systems with mirrors and filters that may be largein size, heavy, and sensitive to shock and vibration. Moreover,human-portable multispectral systems are also usually limited in thenumber of simultaneously detectable wavebands.

Various conventional gas emission detection techniques exist. Forexample, in one approach, the pressure of a system containing gas can bemeasured, and gas leakage may be detected by a decrease in pressure.However, such an approach may only work well if the pressure is keptconstant and the gas leakage is significant, as it may be difficult todetermine whether pressure changes are due to gas leaks, normaloperation, or environmental changes such as temperature increases.Moreover, if the system containing the gas is large (e.g., a long pipe),it may also be difficult to locate the exact position of the leak.

For some gases, sensors may be used to detect the presence of the gas,such as conventional “sniffers” used to detect propane or natural gasleaks. However, such sensors are generally unsuitable for remotemonitoring, as they typically must be in direct contact with thedetected gas.

In another approach, a gas may be remotely sensed using a sensor thatdetects irradiation in or more of the absorption bands (e.g., spectralbands) of the gas. For example, FIG. 8A illustrates transmission as afunction of wavelength for a gas, and FIG. 8B illustrates transmissionthrough the atmosphere as a function of wavelength for an atmosphericcondition.

For gases with absorption bands in the LWIR wavebands, a bandpass filtermay be used. For example, the filter may be tuned such that it closelymatches the absorption band of the gas. The bandpass filter may reducethe amount of irradiance that may be measured by the sensor to a fewpercent of what would be measured if the bandpass filter was notpresent. If the gas is present, it may absorb a significant amount ofthe total irradiance and an operator viewing an image provided by thesensor may be able detect the gas when it occludes the background of theimage (e.g., causing a loss of signal of the background).

However, for such an approach to be effective, the one or more narrowwavebands absorbed by the gas must make up a significant amount of thetotal irradiance being measured by the sensor. For example, for aninfrared sensor sensitive to electromagnetic radiation wavelengths inthe range of 7-13 μm, the absorption band of the detected gas mayrepresent only a fraction of a percent of the total irradiance detectedby the sensor under typical imaging conditions. As a result, the gas mayappear transparent to the sensor because most of the available signalfrom other objects in the background behind the gas may not be absorbedby the gas.

To improve the sensitivity of such sensors, a narrow bandpass filter maybe provided that includes the gas absorption band. In this case, the gasmay absorb a large percentage of electromagnetic radiation in the narrowband which makes the gas easier to detect (e.g., there may be a greaterrelative difference between an image captured when gas is present and animage captured when gas is not present). However, such an approach mayrequire the sensor to be highly responsive. For example, if 95 percentof the signal is lost due to the bandpass filter, then the sensor mayneed to be 20 times more sensitive to preserve scene fidelity. Suchhighly sensitive sensors may require very high performance imagingsystems (e.g., in the infrared waveband, such systems may usecryogenically cooled sensors with large aperture optics). As a result,such systems may be two orders of magnitude more expensive than uncooledinfrared systems, may have significant power requirements which makethem unsuitable for battery operation, and may be larger and heavierthan uncooled systems.

Moreover, some materials (e.g., other than gas) may have spectralproperties that match that of the gas being detected such that theirradiance from such materials mostly fall within the absorption band ofthe gas. This may cause false alarms. For example, some surfaces such aspainted surfaces or metals may emit very little signal in a bandmatching that of one of the gas absorption bands. One way to reduce suchfalse alarms is to measure irradiance in multiple spectral bands thatmatch multiple absorption bands of the gas. This can be accomplished bydesigning a spectral filter that has significant transmission in morethan one spectral band. However, this may complicate the design of thefilter and may limit its efficiency (e.g., as measured by the percent ofirradiance transmitted to the sensor). Another way to reduce such falsealarms is to time multiplex multiple filters having different spectralbands (e.g., by using a rotating filter wheel and a single detector).

However, such an approach may require registering images to compensatefor imager or scene motion (e.g., misalignment may be introduced whenimaging non static scenes or when the imager is hand held or otherwisemoving). In addition, such an approach may only allow for shortintegration or exposure times that are set by the period of the filterwheel rotations.

FIG. 8C illustrates a process of performing gas detection in accordancewith an embodiment of the disclosure. As discussed, different sensorarrays 202 may detect different broad or narrow bands (e.g., wavelengthranges) of electromagnetic radiation. Accordingly, sensor array signals801 (e.g., signals, samples, or data values, such as pixel values,provided by various sensor arrays 202 in response to detectedelectromagnetic radiation) may be provided that correspond to differentbands. In one embodiment, some sensor arrays 202 may be configured todetect broad spectral bands (BSB) of electromagnetic radiation, andother sensor arrays 202 may be configured to detect narrow spectralbands (NSB) of electromagnetic radiation. For example, the NSB mayapproximately match one of the absorption bands (e.g., wavelengthranges) of a known gas (e.g., as shown in FIG. 8A). In one embodiment,NSBs may include all or portions of various wavebands, such as thermalradiation, LWIR radiation, MWIR radiation, SWIR radiation, NIRradiation, visible light (VIS), and/or other ranges. In one embodiment,BSBs may include wavebands greater than that of such NSBs.

Lenses 208 associated with the BSB or NSB sensor arrays 202 may becoated or otherwise filtered to reflect most electromagnetic radiationoutside of their respective bands. Therefore, sensor array signals 801may include some signals corresponding to BSB electromagnetic radiationand some signals corresponding to NSB electromagnetic radiation.

In block 802, sensor array signals 801 are processed to determinewhether they correspond to a BSB sensor array or an NSB sensor array. Inthis regard, samples corresponding to BSB sensor array signals arepassed to block 803 where the samples are mapped to a global BSB scenecoordinate space. In this regard, each sensor 206 and its correspondingpixel of each BSB sensor array 202 may be mapped to a correspondingcoordinate of the BSB scene coordinate space, for example, by selectinga scene coordinate (e.g., pixel) having a center that closest matchesthe center of the corresponding sensor 206. Samples corresponding to NSBsensor array signals are passed to block 804 where the samples aremapped to a global NSB scene coordinate space. In this regard, eachsensor 206 and its corresponding pixel of each NSB sensor array 202 maybe mapped to a corresponding coordinate of the NSB scene coordinatespace, for example, by selecting a scene coordinate having a center thatclosest matches the center of the corresponding sensor 206.

At block 805, the mapped samples (e.g., pixel values) provided by theBSB sensor arrays 202 for particular scene coordinates are compared withthe mapped samples (e.g., pixel values) provided by the NSB sensorarrays 202 for the same scene coordinates. For example, in oneembodiment, the NSB may be a subset of the BSB. In this case, ifapproximately 5 percent of the irradiance measured by the BSB sensorarrays 202 is attributable to NSB electromagnetic radiation, then it maybe expected that the signal provided by an NSB sensor array 202 for aparticular scene coordinate may correspond to approximately 5 percent ofthe signal provided by a BSB sensor array 202 for the same scenecoordinate.

Therefore, if mapped sample value provided by the NSB sensor array 202is close to zero or at least much lower than 5 percent of the mappedsample value provided by the BSB sensor array 202 for the same scenecoordinate, then such values may indicate that a gas is present at thescene coordinate (block 807) (e.g., a gas is absorbing NSBelectromagnetic radiation at the scene coordinate). The presence of thegas may be indicated at the scene coordinate by processing the mappedsamples (block 808) using spectral transmission data for BSB and NSBsensor arrays (block 806) to provide an image 809 (e.g., a result image)that is, for example, highlighted or color coded at the scenecoordinates corresponding to the identified gas.

In another embodiment, different NSB sensor arrays 202 may detect NSBelectromagnetic radiation in different narrow bands. For example, afirst group of one or more NSB sensor arrays 202 may detect NSBelectromagnetic radiation in a first narrow band, and a second group ofone or more NSB sensor arrays 202 may detect NSB electromagneticradiation in a second narrow band that differs from the first narrowband. Additional groups of NSB sensor arrays 202 associated with othernarrow bands may also be provided.

Gases may be detected with high accuracy using different NSB sensorarrays 202 directed toward different NSBs. For example, the differentNSBs may be associated with different absorption bands of the same gas.Thus, by using such different NSB sensor arrays 202 in the process ofFIG. 8C, sample values (e.g., signal strength) provided by BSB sensorarrays 202 may be compared with sample values provided by different NSBsensor arrays 202 for different NSBs. Thus, if a gas has multipleabsorption bands, then the detection of such bands using the differentNSBs may increase the accuracy of gas detection and reduce thelikelihood of false detections (e.g., due to multiple gases or materialssharing an identical or similar absorption band).

In another embodiment, one or more NSB sensor arrays 202 may detect NSBelectromagnetic radiation in multiple narrow bands that match theabsorption bands of multiple gases. In this case, multiple gases withdifferent spectral properties may be detected.

Moreover, any of the described approaches using NSB sensor arrays 202may be combined as desired. For example, one or more multiple NSB sensorarrays 202 may be used to detect multiple NSBs for a single gas ordifferent gases. Advantageously, the use of NSB sensor arrays 202 maypermit multiple types of gases to be detected by a single imager array200.

In various embodiments, the features of imager array 200 may be appliedto other implementations. For example, FIG. 9A illustrates an imagerarray 900 including a plurality of sensor arrays 902/912 and abeamsplitter 901 in accordance with an embodiment of the disclosure. Forexample, in one embodiment, sensor array 902 may include sensors 906(e.g., InGaAs sensors) and a lens 908 provided by a LWIR camera, andsensor array 912 may include sensors 916 (e.g., InGaAs sensors) and alens 918 be provided by a VIS/NIR camera. In this regard, two camerasmay be used in the illustrated embodiment if, for example, no suitabledetector material is available that is sensitive to all wavebands ofinterest. Advantageously, LWIR and VIS/NIR cameras may be implementedwithout requiring the extra weight and size of associated coolingequipment.

As shown in FIG. 9A, electromagnetic radiation from scene 170 may passthrough a common shared aperture 903 to beamsplitter 901 which passes orreflects the electromagnetic radiation to sensor arrays 902 and 912. Theuse of beamsplitter 901 and shared aperture 903 may minimize parallaxbetween the two cameras. Although only two sensor arrays 902/912 areidentified in FIG. 9A, it will be appreciated that any desired number ofsensor arrays 902/912 or cameras may be used.

Because phase shift between sensor arrays 902 and 916 may be used toprovide various features (e.g., in accordance with embodiments describedin this disclosure), sensor arrays 902 and 916 need not be preciselyaligned with each other. This reduces the need for a complexboresighting process and mechanism in this embodiment.

FIG. 9B illustrates an imager array 920 including a plurality of cameras922 in accordance with an embodiment of the disclosure. In thisembodiment, individual cameras 922 may be used in place of individualsensor arrays 202. Although only two cameras 922 are identified in FIG.9B, it will be appreciated that any desired number of cameras 922 may beused.

Imagers sensitive to radiation in the infrared waveband usually haveonly a small number of sensors compared to imagers sensitive toradiation in the visible wave band. This is due to various reasons suchas, for example, the larger aperture optics and larger sensor elementstypically used for infrared radiation, as well as the cost of materials(e.g., germanium and silicon) used for infrared optics.

In one embodiment, an artificial neural network (ANN) may be used toestimate high resolution images from low resolution images provided bysensor arrays 202. Such high resolution images may be used, for example,for target tracking or other applications.

An ANN may be used to implement a nonlinear classification process inwhich nonlinear, scene-dependent, and wavelength-dependent relationshipsare mapped between low spatial frequency signals (e.g., low resolutionpixel values captured of a scene) to corresponding high spatialfrequency signals (e.g., high resolution pixel values stored in adatabase for scenes previously imaged at high resolution). For example,one or more such ANNs may be implemented to perform radial basisfunction (RBF) processing techniques which may be suitable for hardware(e.g., using digital circuitry, analog ROIC circuitry, or othercircuitry) or software implementations.

In one embodiment, the ANN may include individual artificial neurons(e.g., modeled on a brain) that are implemented by individual processorsin a completely parallel architecture. In this case, all such processorsmay be configured to access data simultaneously and provide output datawithin several clock cycles.

FIG. 10 illustrates a process of providing a high resolution image usingan ANN in accordance with an embodiment of the disclosure. For example,in one embodiment, the ANN may be provided by appropriate processors,memories, and machine readable instructions of imaging system 100.

In block 1002, imaging system 100 captures high resolution trainingimages of a desired type of scene 170. For example, imaging system 1002may use a separate high resolution imager array, or may configure imagerarray 200 for high resolution operation (e.g., by using all of sensorarrays 202 as a single sensor array). In one embodiment, the trainingperformed in block 1002 may be performed non-iteratively may greatlyimprove the real time possibilities for online performance enhancements(e.g., continuous scene learning).

The high resolution training images may be stored, for example, in anappropriate memory of imaging system 100, a local or remote database, orany other desired location. Thus, imaging system 100 may have access toa set of high resolution training images (e.g., a learned “dictionary”)of a particular type of scene 170 which may be subsequently imaged bylow resolution sensor arrays 202 of imager array 200.

In block 1004, imaging system 100 captures one or more low resolutionimages of a particular scene 170 using low resolution sensor arrays 202.In block 1006, imaging system 100 processes individual pixels (e.g.,using an ANN in accordance with RBF techniques) of the low resolutionimage to determine a mapping from each pixel to at least a portion ofone or more of the high resolution training images.

In one embodiment, the scene imaged in block 1004 should be at leastsimilar to the scene imaged in block 1002 in order to increase thelikelihood of accurate pixel mappings in block 1006. For example, if thescenes are significantly different, then imaging system 100 may flag thelow resolution scene images as invalid.

In block 1008, imaging system 100 replaces the pixels of the lowresolution image with the mapped high resolution training images toprovide a resulting high resolution image (e.g., a result image).

In one embodiment, the process of FIG. 10 may permit low resolutionimages to be converted into high resolution images on par or better thanwould be available with a conventional single aperture imager. Forexample, in one embodiment, the process of FIG. 10 may convert lowresolution images of approximately 80 by 80 pixels up to high resolutionimages.

Moreover, because the process of FIG. 10 does not require thecalculation of a per pixel optical flow, the computational burden may bekept reasonable and well within the processing available in a customsystem on a chip (SoC) device.

In another embodiment, the process of FIG. 10 may be modified inaccordance with various alternate process steps. For example, in block1002, low resolution training images (e.g., low resolution versions ofthe high resolution training images) may also be provided. In oneembodiment, the high resolution training images may be captured by ahigh resolution sensor array (e.g., sensor array 232) and the lowresolution training images may be captured by a low resolution sensorarray (e.g., sensor array 202).

Such high and low resolution training images may be capturedsubstantially simultaneously using different sensor arrays 232 and 202,where a high resolution training image may have a narrow FoV containedwithin the imaging cone of a wide FoV of a low resolution trainingimage. For example, if the image cone identified by twice the angle ρ iswithin the image cone identified by twice the angle Φ (see FIG. 2D), andthe images provided by sensor arrays 202 and 232 are accurately mappedto a common scene coordinate space, the low resolution training imagemay be provided directly from the image captured by sensor array 202(e.g., if the low resolution is half the high resolution, then 8 by 8pixels from imager array 232 may be represented as 4 by 4 pixels fromimager array 202).

In another embodiment, the high and low resolution training images maybe captured using identical sensor arrays with different optics toprovide different narrow and wide FoVs. In another embodiment, the highand low resolution training images may be captured at different timesusing a single optic capable of using at least two differentmagnification settings (e.g., different zoom positions). In anotherembodiment, a low resolution training image may be created by blurring ahigh resolution training image and resampling at a lower resolutiondensity.

The high and low resolution training images provided in block 1002 maybe separated into a plurality of sub images before being stored in adatabase. For example, in one embodiment, each low resolution trainingsub image may provide 8 by 8 pixels. Also in block 1002, the lowresolution training sub images may be mapped to their corresponding highresolution sub images.

In block 1006, low resolution images (e.g., non-training images capturedin block 1004) may be separated into a plurality of sub images, and thesub images may be mapped to previously stored low resolution trainingsub images. For example, in one embodiment, such mapping may be based ona Euclidian distance calculated between a vector defined by pixel valuesof the non-training low resolution sub images and a vector defined bypixel values of the training low resolution sub images.

Also in block 1006, the low resolution training sub images that havebeen mapped to the non-training low resolution sub images may be used toidentify high resolution sub images (e.g., previously mapped to the lowresolution training sub images in block 1002). As a result, theidentified high resolution sub images may be used to replace thenon-training low resolution sub images in block 1008 to provide a highresolution image (e.g., a result image).

In other embodiments, motion free, sample based, or single image superresolution processing techniques may be used with imager array 200. Forexample, such techniques may rely upon a learned database (e.g.,dictionary) of high resolution images (e.g., samples), and the quality(e.g., measured by a peak signal to noise ratio (PSNR)) of a superresolved result image may depend significantly on the similarity betweenimages in the database and the imaged scene 170. Therefore, the qualityof result images obtained using such techniques may be improved if thehigh resolution images in the database are images of the actual imagedscene 170.

In accordance with various embodiments, imager array 200 may be usedand/or modified for use in a variety of other applications. For example,in one embodiment, imaging system 100 may process images provided byvarious sensor arrays 202 to simultaneously provide the images to a useras well as perform signature correlation to perform, for example, lasertargeting, automated target detection and tracking, or other operations.

In other embodiments, imager array 200 may be used in variousapplications such as, for example, night vision goggles, ballisticmounted detection and tracking, autonomous vehicle payloads, and others.In one embodiment, imager array 200 may be implemented with a relativelysmall size and a substantially flat profile that permits convenientintegration into clothing, helmets, or other installations. For example,a cube implemented with six imager arrays 200 (e.g., one imager array200 per plane of the cube) may be used to provide full sphericalimaging.

In other embodiments, imager array 200 may be used in variousconsumer-oriented applications where low cost, multispectral, infrared,or other types of imaging systems may be useful. In another embodiment,imager array 200 may be used to perform automatic calibration inradiometric applications by taking into account emissivity in differentwavebands.

FIGS. 11A-F illustrate several views of types of imager arrays having aplurality of infrared imaging modules 1102 in accordance withembodiments of the disclosure. Imager arrays 1100 a-e of FIGS. 11A-F maybe used, for example, to implement image capture component 130 ofimaging system 100. In particular, embodiments of infrared imagingmodule 2100 discussed in connection with FIGS. 12-24, below, may be usedto provide imaging modules 1102, for example, in place of (e.g., may beindividually interchanged with) any one or all of the sensor arraysdiscussed in connection with FIGS. 1-10. In some embodiments, any one ofthe infrared imaging modules 1102 in imager arrays 1100 a-e of FIGS.11A-F may be individually interchanged with one or more of the sensorarrays discussed in connection with FIGS. 1-10.

In various embodiments, infrared imaging modules 1102 may each include amodule housing 1120 (e.g., housing 2120 in FIG. 14), an optical element1108 (e.g., optical element 2180 in FIG. 14) fixed relative to themodule housing 1120, and a plurality of infrared sensors in a focalplane array adapted to capture an image based on infrared radiationreceived through optical element 1108. In some embodiments, opticalelement 1108 may be at least partially enclosed by a lens barrel 1109(e.g., lens barrel 2110 in FIG. 14 which may integrated with or formedseparately from housing 2120). In some embodiments, infrared imagingmodules 1102 may include other elements or different elements, such asthose described in connection with FIGS. 12-24.

In various embodiments, infrared imaging modules 1102 may perform multispectral imaging to selectively detect desired ranges of infraredradiation, such as thermal radiation, long wave infrared (LWIR)radiation, mid wave infrared (MWIR) radiation, short wave infrared(SWIR) radiation, near infrared (NIR) radiation, and/or other ranges. Inthis regard, optical elements 1108 may include appropriate coatings, orinfrared imaging modules 1102 may be provided with appropriate filters,to filter the infrared radiation received by infrared sensors (e.g.,infrared sensors 2132 of FIG. 14) of infrared imaging modules 1102. As aresult, different infrared imaging modules 1102 may detect differentbroad or narrow bands of electromagnetic (e.g., in particular, infrared)radiation.

In various embodiments, one or more infrared imaging modules 1102 may beimplemented with substantially equal sizes and/or different sizes in thesame or similar fashion as sensor arrays to provide the features andadvantages of such sizing as described herein.

Infrared imaging modules 1102 may be arranged in various configurationswithin imager arrays. Such configurations may include, for example, asquare lattice, a rectangular lattice, an oblique lattice, a rhombiclattice, a hexagonal lattice, or any other configuration or combinationof configurations, for example. In some embodiments, module housings1120 of infrared imaging modules 1102 may be configured to complement aparticular configuration, such as being predominantly triangular orrectangular, for example.

In related embodiments, particular configurations may be chosen based ona type of image processing to be performed on image data captured byimager arrays, for example, or based on a desired ratio of a number ofinfrared imaging modules 1102 to a two dimensional area of an imagerarray (e.g., a packing ratio). In further embodiments, increasing apacking ratio of an imager array may effectively shorten and/orhomogenize a distance between optical axes of adjoining ones of infraredimaging modules 1102. In such embodiments, an effective resolution ofimager array 1100 a may be increased. In some embodiments, a packingratio of an imager array may be adjusted by staggering infrared imagingmodules 1102 in a multi-level staggered configuration, for example.

Although various components of infrared imaging modules 1102, such asoptical elements 1108, lens barrels 1109, and module housings 1120 aregenerally shown as being substantially similar in FIGS. 11A-F, theseand/or other components may be implemented differently in variousconfigurations, such as in different configurations of imager arrays.

As shown in the top view provided by FIG. 11A, imager array 1100 a mayinclude an array (e.g., 8 by 6 in one embodiment) of infrared imagingmodules 1102 a arranged in a square lattice configuration. Also shownare optional partitions 1104 a, which may provide structural support forimager array 1100 a, for example, and/or may be used to limit a field ofview of one or more infrared imaging modules 1102 a. In someembodiments, infrared imaging modules 1102 a may be used to performmulti spectral imaging as described herein. For example, in oneembodiment, at least four spectral bands may be detected, depicted inFIG. 11A as groups 1120, 1122, 1124, and 1126 of infrared imagingmodules 1102 a.

FIG. 11B shows imager array 1100 a viewed along section lines 11B-11B inFIG. 11A. In some embodiments, imager array 1100 a may include a base1110 a for structural support and/or electrical routing, for example.Distance 1111 a indicates a height of infrared imaging modules 1102 afrom base 1110 a, and distance 1112 a indicates a height of partitions1104 a from base 1110 a. Distances 1111 a and 1112 a may be selected toproduce a desired field of view 1106 a (e.g., an optical width) for eachof infrared imaging modules 1102 a, for example. In some embodiments,imager array 1100 a may not include optional partitions 1104 a, andinstead may rely on dimensions and configurations of infrared imagingmodules 1102 a to produce a desired field of view, including aselectable field of view, for example, for each of infrared imagingmodules 1102 a.

FIGS. 11C-D illustrate an embodiment where imager array 11 OOc includesinfrared imaging modules 1102 arranged in a two-level staggeredconfiguration with infrared imaging modules 1102 c arranged in an upperlevel of imager array 1100 c and infrared imaging modules 1102 darranged in a lower level of imager array 1100 c. In embodiments whereinfrared imaging modules 1102 c and 1102 d are similar in size toinfrared imaging modules 1102 a of FIGS. 11A-B, imager array 1100 c hasa larger packing ratio than imager array 1100 a. Infrared imagingmodules 1102 c-d may include optical elements 1108, lens barrels 1109,housings 1120, and/or other features as discussed.

FIG. 11D illustrates imager array 1100 c viewed along section lines11D-11D in FIG. 11C. FIG. 11D shows base 1110 c, optional partitions1104 c (e.g., not shown in FIG. 11C for clarity purposes) with opticalwidth 1106 c, and distances 1111 c-1112 c. Distance 1111 c indicates aheight of lower infrared imaging modules 1103 c from base 1110 c, anddistance 1112 c indicates a height of upper infrared imaging modules1102 c from base 1110 c. As shown in FIG. 11D, in some embodiments, oneor more of lower infrared imaging modules 1103 c may include an extendedlens barrel 1109 c and/or an extended optical element (e.g., withinextended lens barrel 1109 c), for example, which may be adapted toapproximate a height of upper infrared imaging modules 1102 c and, insome embodiments, adjust a field of view of lower infrared imagingmodules 1103 c to match that of upper infrared imaging modules 1102 c.

As can be seen from FIGS. 11C-D, imager array 1100 c may be implementedin a multi-level staggered configuration with two layers of infraredimaging modules 1102 c and 1102 d, where either columns or rows ofimager array 1100 c may be alternatingly staggered in height in order toincrease an overall packing ratio of imager array 1100 c. Although FIG.11C shows imager array 1100 c with infrared imaging modules 1102 c and1102 d arranged in a square lattice configuration, in other embodiments,imager array 1100 c may include a plurality of infrared imaging modules1102 c and 1102 d arranged in a different lattice configuration, forexample, and rows, columns, or other groupings of the infrared imagingmodules may be arranged in a two-level staggered configuration adaptedto increase a packing ratio of imager array 1100 c. In some embodiments,housings 1120 and/or other appropriate components of infrared imagingmodules 1102 c and 1102 d may be configured to complement a particularlattice configuration and/or multi-level staggered configuration, suchas being predominantly triangular or rectangular, for example, and/orbeing notched or sloped to interlock with adjacent upper imaging modules1102 c and/or lower infrared imaging modules 1102 d.

FIGS. 11E-F illustrate an embodiment where imager array 1100 e includesinfrared imaging modules 1102 e-1102 h arranged in a four-levelstaggered configuration. In embodiments where infrared imaging modules1102 e-1102 h are similar in size to infrared imaging modules 1102 c and1102 d, imager array 1100 e has a larger packing ratio than imager array1100 c. More generally, imager arrays may include infrared imagingmodules arranged in a plurality of levels that are staggered in order toincrease a packing ratio of the imager array, for example, or tofacilitate a particular image processing technique. Such imageprocessing techniques may include types and/or methods of Fouriertransforms, interpolation methods, and color (e.g., pseudo-color, orinfrared spectrum) distribution methods, for example. In someembodiments, imager array 1100 e may be implemented with optionalpartitions (e.g., not shown in FIGS. 11E-F for clarity purposes) assimilarly described herein.

FIGS. 11E-F illustrate an embodiment where imager array 1100 e includesinfrared imaging modules 1102 e arranged in an first level of imagerarray 1100 e, infrared imaging modules 1102 f arranged in a second levelof imager array 1100 e, infrared imaging modules 1102 g arranged in athird level of imager array 1100 e, and infrared imaging modules 1102 harranged in a fourth level of imager array 1100 e Infrared imagingmodules 1102 e-h may include optical elements 1108, lens barrels 1109,extended lens barrels 1109 e (e.g., as discussed in relation to FIG.11D), extended optical elements (e.g., within extended lens barrels 1109e), housings 1120, and/or other features as discussed. FIG. 11F showsimager array 1100 e viewed along section lines 11F-11F in FIG. 11E andincludes base 1110 e and distances 1111 e-1114 e. Distance 1111 eindicates a height of first level infrared imaging modules 1102 e frombase 1110 e, distance 1112 e indicates a height of second level infraredimaging modules 1102 f from base 1110 e, distance 1111 e indicates aheight of third level infrared imaging modules 1102 g from base 1110 e,and distance 1114 e indicates a height of fourth level infrared imagingmodules 1102 h from base 1110 e. In some embodiments, one or more of thelower level infrared imaging modules 1102 e-g may include extendedoptical elements and/or extended lens barrels as similarly discussedwith regard to FIG. 11D to approximate a height of fourth level (e.g.,top level) infrared imaging modules 1102 h.

As can be seen from FIGS. 11E-F, imager array 1100 e may be implementedin a multi-level staggered configuration with four layers of infraredimaging modules 1102 e-h, where infrared imaging modules of imager array1100 e may be staggered across four levels in height in order toincrease an overall packing ratio of imager array 1100 e. Although FIG.11E shows imager array 1100 e with infrared imaging modules 1102 e-harranged in a square lattice configuration, in other embodiments, imagerarray 1100 e may include a plurality of infrared imaging modules 1102e-h arranged in a different lattice configuration, for example, andother groupings of the infrared imaging modules may be arranged in afour-level staggered configuration adapted to increase a packing ratioof imager array 1100 e. More generally, infrared imaging modules of animager array may be arranged in a multi-level staggered configurationadapted to increase a packing ratio of the imager array. In variousembodiments, module housings 1120 of infrared imaging modules 1102 e-hmay be configured to complement a particular lattice configurationand/or multi-level staggered configuration.

FIG. 12 illustrates an infrared imaging module 2100 (e.g., an infraredcamera or an infrared imaging device) configured to be implemented in ahost device 2102 in accordance with an embodiment of the disclosureInfrared imaging module 2100 may be implemented, for one or moreembodiments, with a small form factor and in accordance with wafer levelpackaging techniques or other packaging techniques.

In one embodiment, infrared imaging module 2100 may be configured to beimplemented in a small portable host device 2102, such as a mobiletelephone, a tablet computing device, a laptop computing device, apersonal digital assistant, a visible light camera, a music player, orany other appropriate mobile device. In this regard, infrared imagingmodule 2100 may be used to provide infrared imaging features to hostdevice 2102. For example, infrared imaging module 2100 may be configuredto capture, process, and/or otherwise manage infrared images and providesuch infrared images to host device 2102 for use in any desired fashion(e.g., for further processing, to store in memory, to display, to use byvarious applications running on host device 2102, to export to otherdevices, or other uses).

In various embodiments, infrared imaging module 2100 may be configuredto operate at low voltage levels and over a wide temperature range. Forexample, in one embodiment, infrared imaging module 2100 may operateusing a power supply of approximately 2.4 volts, 2.5 volts, 2.8 volts,or lower voltages, and operate over a temperature range of approximately−20 degrees C. to approximately +60 degrees C. (e.g., providing asuitable dynamic range and performance over an environmental temperaturerange of approximately 80 degrees C.). In one embodiment, by operatinginfrared imaging module 2100 at low voltage levels, infrared imagingmodule 2100 may experience reduced amounts of self heating in comparisonwith other types of infrared imaging devices. As a result, infraredimaging module 2100 may be operated with reduced measures to compensatefor such self heating.

As shown in FIG. 12, host device 2102 may include a socket 2104, ashutter 2105, motion sensors 2194, a processor 2195, a memory 2196, adisplay 2197, and/or other components 2198. Socket 2104 may beconfigured to receive infrared imaging module 2100 as identified byarrow 2101. In this regard, FIG. 13 illustrates infrared imaging module2100 assembled in socket 2104 in accordance with an embodiment of thedisclosure.

Motion sensors 2194 may be implemented by one or more accelerometers,gyroscopes, or other appropriate devices that may be used to detectmovement of host device 2102. Motion sensors 2194 may be monitored byand provide information to processing module 2160 or processor 2195 todetect motion. In various embodiments, motion sensors 2194 may beimplemented as part of host device 2102 (as shown in FIG. 12), infraredimaging module 2100, or other devices attached to or otherwiseinterfaced with host device 2102.

Processor 2195 may be implemented as any appropriate processing device(e.g., logic device, microcontroller, processor, application specificintegrated circuit (ASIC), or other device) that may be used by hostdevice 2102 to execute appropriate instructions, such as softwareinstructions provided in memory 2196. Display 2197 may be used todisplay captured and/or processed infrared images and/or other images,data, and information. Other components 2198 may be used to implementany features of host device 2102 as may be desired for variousapplications (e.g., clocks, temperature sensors, a visible light camera,or other components). In addition, a machine readable medium 2193 may beprovided for storing non-transitory instructions for loading into memory2196 and execution by processor 2195.

In various embodiments, infrared imaging module 2100 and socket 2104 maybe implemented for mass production to facilitate high volumeapplications, such as for implementation in mobile telephones or otherdevices (e.g., requiring small form factors). In one embodiment, thecombination of infrared imaging module 2100 and socket 2104 may exhibitoverall dimensions of approximately 8.5 mm by 8.5 mm by 5.9 mm whileinfrared imaging module 2100 is installed in socket 2104.

FIG. 14 illustrates an exploded view of infrared imaging module 2100juxtaposed over socket 2104 in accordance with an embodiment of thedisclosure. Infrared imaging module 2100 may include a lens barrel 2110,a housing 2120, an infrared sensor assembly 2128, a circuit board 2170,a base 2150, and a processing module 2160.

Lens barrel 2110 may at least partially enclose an optical element 2180(e.g., a lens) which is partially visible in FIG. 14 through an aperture2112 in lens barrel 2110. Lens barrel 2110 may include a substantiallycylindrical extension 2114 which may be used to interface lens barrel2110 with an aperture 2122 in housing 2120.

Infrared sensor assembly 2128 may be implemented, for example, with acap 2130 (e.g., a lid) mounted on a substrate 2140. Infrared sensorassembly 2128 may include a plurality of infrared sensors 2132 (e.g.,infrared detectors) implemented in an array or other fashion onsubstrate 2140 and covered by cap 2130. For example, in one embodiment,infrared sensor assembly 2128 may be implemented as a focal plane array(FPA). Such a focal plane array may be implemented, for example, as avacuum package assembly (e.g., sealed by cap 2130 and substrate 2140).In one embodiment, infrared sensor assembly 2128 may be implemented as awafer level package (e.g., infrared sensor assembly 2128 may besingulated from a set of vacuum package assemblies provided on a wafer).In one embodiment, infrared sensor assembly 2128 may be implemented tooperate using a power supply of approximately 2.4 volts, 2.5 volts, 2.8volts, or similar voltages.

Infrared sensors 2132 may be configured to detect infrared radiation(e.g., infrared energy) from a target scene including, for example, midwave infrared wave bands (MWIR), long wave infrared wave bands (LWIR),and/or other thermal imaging bands as may be desired in particularimplementations. In one embodiment, infrared sensor assembly 2128 may beprovided in accordance with wafer level packaging techniques.

Infrared sensors 2132 may be implemented, for example, asmicrobolometers or other types of thermal imaging infrared sensorsarranged in any desired array pattern to provide a plurality of pixels.In one embodiment, infrared sensors 2132 may be implemented as vanadiumoxide (VOx) detectors with a 17 μm pixel pitch. In various embodiments,arrays of approximately 32 by 32 infrared sensors 2132, approximately 64by 64 infrared sensors 2132, approximately 80 by 64 infrared sensors2132, or other array sizes may be used.

Substrate 2140 may include various circuitry including, for example, aread out integrated circuit (ROIC) with dimensions less thanapproximately 5.5 mm by 5.5 mm in one embodiment. Substrate 2140 mayalso include bond pads 2142 that may be used to contact complementaryconnections positioned on inside surfaces of housing 2120 when infraredimaging module 2100 is assembled as shown in FIG. 14. In one embodiment,the ROIC may be implemented with low-dropout regulators (LDO) to performvoltage regulation to reduce power supply noise introduced to infraredsensor assembly 2128 and thus provide an improved power supply rejectionratio (PSRR). Moreover, by implementing the LDO with the ROIC (e.g.,within a wafer level package), less die area may be consumed and fewerdiscrete die (or chips) are needed.

FIG. 15 illustrates a block diagram of infrared sensor assembly 2128including an array of infrared sensors 2132 in accordance with anembodiment of the disclosure. In the illustrated embodiment, infraredsensors 2132 are provided as part of a unit cell array of a ROIC 2402.ROIC 2402 includes bias generation and timing control circuitry 2404,column amplifiers 2405, a column multiplexer 2406, a row multiplexer2408, and an output amplifier 2410. Image frames (e.g., thermal images)captured by infrared sensors 2132 may be provided by output amplifier2410 to processing module 2160, processor 2195, and/or any otherappropriate components to perform various processing techniquesdescribed herein. Although an 8 by 8 array is shown in FIG. 15, anydesired array configuration may be used in other embodiments. Furtherdescriptions of ROICs and infrared sensors (e.g., microbolometercircuits) may be found in U.S. Pat. No. 6,028,309 issued Feb. 22, 2000,which is incorporated herein by reference in its entirety.

Infrared sensor assembly 2128 may capture images (e.g., image frames)and provide such images from its ROIC at various rates. Processingmodule 2160 may be used to perform appropriate processing of capturedinfrared images and may be implemented in accordance with anyappropriate architecture. In one embodiment, processing module 2160 maybe implemented as an ASIC. In this regard, such an ASIC may beconfigured to perform image processing with high performance and/or highefficiency. In another embodiment, processing module 2160 may beimplemented with a general purpose central processing unit (CPU) whichmay be configured to execute appropriate software instructions toperform image processing, coordinate and perform image processing withvarious image processing blocks, coordinate interfacing betweenprocessing module 2160 and host device 2102, and/or other operations. Inyet another embodiment, processing module 2160 may be implemented with afield programmable gate array (FPGA). Processing module 2160 may beimplemented with other types of processing and/or logic circuits inother embodiments as would be understood by one skilled in the art.

In these and other embodiments, processing module 2160 may also beimplemented with other components where appropriate, such as, volatilememory, non-volatile memory, and/or one or more interfaces (e.g.,infrared detector interfaces, inter-integrated circuit (I2C) interfaces,mobile industry processor interfaces (MIPI), joint test action group(JTAG) interfaces (e.g., IEEE 1149.1 standard test access port andboundary-scan architecture), and/or other interfaces).

In some embodiments, infrared imaging module 2100 may further includeone or more actuators 2199 which may be used to adjust the focus ofinfrared image frames captured by infrared sensor assembly 2128. Forexample, actuators 2199 may be used to move optical element 2180,infrared sensors 2132, and/or other components relative to each other toselectively focus and defocus infrared image frames in accordance withtechniques described herein. Actuators 2199 may be implemented inaccordance with any type of motion-inducing apparatus or mechanism, andmay positioned at any location within or external to infrared imagingmodule 2100 as appropriate for different applications.

When infrared imaging module 2100 is assembled, housing 2120 maysubstantially enclose infrared sensor assembly 2128, base 2150, andprocessing module 2160. Housing 2120 may facilitate connection ofvarious components of infrared imaging module 2100. For example, in oneembodiment, housing 2120 may provide electrical connections 2126 toconnect various components as further described.

Electrical connections 2126 (e.g., conductive electrical paths, traces,or other types of connections) may be electrically connected with bondpads 2142 when infrared imaging module 2100 is assembled. In variousembodiments, electrical connections 2126 may be embedded in housing2120, provided on inside surfaces of housing 2120, and/or otherwiseprovided by housing 2120. Electrical connections 2126 may terminate inconnections 2124 protruding from the bottom surface of housing 2120 asshown in FIG. 14. Connections 2124 may connect with circuit board 2170when infrared imaging module 2100 is assembled (e.g., housing 2120 mayrest atop circuit board 2170 in various embodiments). Processing module2160 may be electrically connected with circuit board 2170 throughappropriate electrical connections. As a result, infrared sensorassembly 2128 may be electrically connected with processing module 2160through, for example, conductive electrical paths provided by: bond pads2142, complementary connections on inside surfaces of housing 2120,electrical connections 2126 of housing 2120, connections 2124, andcircuit board 2170. Advantageously, such an arrangement may beimplemented without requiring wire bonds to be provided between infraredsensor assembly 2128 and processing module 2160.

In various embodiments, electrical connections 2126 in housing 2120 maybe made from any desired material (e.g., copper or any other appropriateconductive material). In one embodiment, electrical connections 2126 mayaid in dissipating heat from infrared imaging module 2100.

Other connections may be used in other embodiments. For example, in oneembodiment, sensor assembly 2128 may be attached to processing module2160 through a ceramic board that connects to sensor assembly 2128 bywire bonds and to processing module 2160 by a ball grid array (BGA). Inanother embodiment, sensor assembly 2128 may be mounted directly on arigid flexible board and electrically connected with wire bonds, andprocessing module 2160 may be mounted and connected to the rigidflexible board with wire bonds or a BGA.

The various implementations of infrared imaging module 2100 and hostdevice 2102 set forth herein are provided for purposes of example,rather than limitation. In this regard, any of the various techniquesdescribed herein may be applied to any infrared camera system, infraredimager, or other device for performing infrared/thermal imaging.

Substrate 2140 of infrared sensor assembly 2128 may be mounted on base2150. In various embodiments, base 2150 (e.g., a pedestal) may be made,for example, of copper formed by metal injection molding (MIM) andprovided with a black oxide or nickel-coated finish. In variousembodiments, base 2150 may be made of any desired material, such as forexample zinc, aluminum, or magnesium, as desired for a given applicationand may be formed by any desired applicable process, such as for examplealuminum casting, MIM, or zinc rapid casting, as may be desired forparticular applications. In various embodiments, base 2150 may beimplemented to provide structural support, various circuit paths,thermal heat sink properties, and other features where appropriate. Inone embodiment, base 2150 may be a multi-layer structure implemented atleast in part using ceramic material.

In various embodiments, circuit board 2170 may receive housing 2120 andthus may physically support the various components of infrared imagingmodule 2100. In various embodiments, circuit board 2170 may beimplemented as a printed circuit board (e.g., an FR4 circuit board orother types of circuit boards), a rigid or flexible interconnect (e.g.,tape or other type of interconnects), a flexible circuit substrate, aflexible plastic substrate, or other appropriate structures. In variousembodiments, base 2150 may be implemented with the various features andattributes described for circuit board 2170, and vice versa.

Socket 2104 may include a cavity 2106 configured to receive infraredimaging module 2100 (e.g., as shown in the assembled view of FIG. 13).Infrared imaging module 2100 and/or socket 2104 may include appropriatetabs, arms, pins, fasteners, or any other appropriate engagement memberswhich may be used to secure infrared imaging module 2100 to or withinsocket 2104 using friction, tension, adhesion, and/or any otherappropriate manner. Socket 2104 may include engagement members 2107 thatmay engage surfaces 2109 of housing 2120 when infrared imaging module2100 is inserted into a cavity 2106 of socket 2104. Other types ofengagement members may be used in other embodiments.

Infrared imaging module 2100 may be electrically connected with socket2104 through appropriate electrical connections (e.g., contacts, pins,wires, or any other appropriate connections). For example, socket 2104may include electrical connections 2108 which may contact correspondingelectrical connections of infrared imaging module 2100 (e.g.,interconnect pads, contacts, or other electrical connections on side orbottom surfaces of circuit board 2170, bond pads 2142 or otherelectrical connections on base 2150, or other connections). Electricalconnections 2108 may be made from any desired material (e.g., copper orany other appropriate conductive material). In one embodiment,electrical connections 2108 may be mechanically biased to press againstelectrical connections of infrared imaging module 2100 when infraredimaging module 2100 is inserted into cavity 2106 of socket 2104. In oneembodiment, electrical connections 2108 may at least partially secureinfrared imaging module 2100 in socket 2104. Other types of electricalconnections may be used in other embodiments.

Socket 2104 may be electrically connected with host device 2102 throughsimilar types of electrical connections. For example, in one embodiment,host device 2102 may include electrical connections (e.g., solderedconnections, snap-in connections, or other connections) that connectwith electrical connections 2108 passing through apertures 2190. Invarious embodiments, such electrical connections may be made to thesides and/or bottom of socket 2104.

Various components of infrared imaging module 2100 may be implementedwith flip chip technology which may be used to mount components directlyto circuit boards without the additional clearances typically needed forwire bond connections. Flip chip connections may be used, as an example,to reduce the overall size of infrared imaging module 2100 for use incompact small form factor applications. For example, in one embodiment,processing module 2160 may be mounted to circuit board 2170 using flipchip connections. For example, infrared imaging module 2100 may beimplemented with such flip chip configurations.

In various embodiments, infrared imaging module 2100 and/or associatedcomponents may be implemented in accordance with various techniques(e.g., wafer level packaging techniques) as set forth in U.S. patentapplication Ser. No. 12/844,124 filed Jul. 27, 2010, and U.S.Provisional Patent Application No. 61/469,651 filed Mar. 30, 2011, whichare incorporated herein by reference in their entirety. Furthermore, inaccordance with one or more embodiments, infrared imaging module 2100and/or associated components may be implemented, calibrated, tested,and/or used in accordance with various techniques, such as for exampleas set forth in U.S. Pat. No. 7,470,902 issued Dec. 30, 2008, U.S. Pat.No. 6,028,309 issued Feb. 22, 2000, U.S. Pat. No. 6,812,465 issued Nov.2, 2004, U.S. Pat. No. 7,034,301 issued Apr. 25, 2006, U.S. Pat. No.7,679,048 issued Mar. 16, 2010, U.S. Pat. No. 7,470,904 issued Dec. 30,2008, U.S. patent application Ser. No. 12/202,880 filed Sep. 2, 2008,and U.S. patent application Ser. No. 12/202,896 filed Sep. 2, 2008,which are incorporated herein by reference in their entirety.

In some embodiments, host device 2102 may include other components 2198such as a non-thermal camera (e.g., a visible light camera or other typeof non-thermal imager). The non-thermal camera may be a small formfactor imaging module or imaging device, and may, in some embodiments,be implemented in a manner similar to the various embodiments ofinfrared imaging module 2100 disclosed herein, with one or more sensorsand/or sensor arrays responsive to radiation in non-thermal spectrums(e.g., radiation in visible light wavelengths, ultraviolet wavelengths,and/or other non-thermal wavelengths). For example, in some embodiments,the non-thermal camera may be implemented with a charge-coupled device(CCD) sensor, an electron multiplying CCD (EMCCD) sensor, acomplementary metal-oxide-semiconductor (CMOS) sensor, a scientific CMOS(sCMOS) sensor, or other filters and/or sensors.

In some embodiments, the non-thermal camera may be co-located withinfrared imaging module 2100 and oriented such that a field-of-view(FoV) of the non-thermal camera at least partially overlaps a FoV ofinfrared imaging module 2100. In one example, infrared imaging module2100 and a non-thermal camera may be implemented as a dual sensor modulesharing a common substrate according to various techniques described inU.S. Provisional Patent Application No. 61/748,018 filed Dec. 31, 2012,which is incorporated herein by reference.

For embodiments having such a non-thermal light camera, variouscomponents (e.g., processor 2195, processing module 2160, and/or otherprocessing component) may be configured to superimpose, fuse, blend, orotherwise combine infrared images (e.g., including thermal images)captured by infrared imaging module 2100 and non-thermal images (e.g.,including visible light images) captured by a non-thermal camera,whether captured at substantially the same time or different times(e.g., time-spaced over hours, days, daytime versus nighttime, and/orotherwise).

In some embodiments, thermal and non-thermal images may be processed togenerate combined images (e.g., one or more processes performed on suchimages in some embodiments). For example, scene-based NUC processing maybe performed (as further described herein), true color processing may beperformed, and/or high contrast processing may be performed.

Regarding true color processing, thermal images may be blended withnon-thermal images by, for example, blending a radiometric component ofa thermal image with a corresponding component of a non-thermal imageaccording to a blending parameter, which may be adjustable by a userand/or machine in some embodiments. For example, luminance orchrominance components of the thermal and non-thermal images may becombined according to the blending parameter. In one embodiment, suchblending techniques may be referred to as true color infrared imagery.For example, in daytime imaging, a blended image may comprise anon-thermal color image, which includes a luminance component and achrominance component, with its luminance value replaced by theluminance value from a thermal image. The use of the luminance data fromthe thermal image causes the intensity of the true non-thermal colorimage to brighten or dim based on the temperature of the object. Assuch, these blending techniques provide thermal imaging for daytime orvisible light images.

Regarding high contrast processing, high spatial frequency content maybe obtained from one or more of the thermal and non-thermal images(e.g., by performing high pass filtering, difference imaging, and/orother techniques). A combined image may include a radiometric componentof a thermal image and a blended component including infrared (e.g.,thermal) characteristics of a scene blended with the high spatialfrequency content, according to a blending parameter, which may beadjustable by a user and/or machine in some embodiments. In someembodiments, high spatial frequency content from non-thermal images maybe blended with thermal images by superimposing the high spatialfrequency content onto the thermal images, where the high spatialfrequency content replaces or overwrites those portions of the thermalimages corresponding to where the high spatial frequency content exists.For example, the high spatial frequency content may include edges ofobjects depicted in images of a scene, but may not exist within theinterior of such objects. In such embodiments, blended image data maysimply include the high spatial frequency content, which maysubsequently be encoded into one or more components of combined images.

For example, a radiometric component of thermal image may be achrominance component of the thermal image, and the high spatialfrequency content may be derived from the luminance and/or chrominancecomponents of a non-thermal image. In this embodiment, a combined imagemay include the radiometric component (e.g., the chrominance componentof the thermal image) encoded into a chrominance component of thecombined image and the high spatial frequency content directly encoded(e.g., as blended image data but with no thermal image contribution)into a luminance component of the combined image. By doing so, aradiometric calibration of the radiometric component of the thermalimage may be retained. In similar embodiments, blended image data mayinclude the high spatial frequency content added to a luminancecomponent of the thermal images, and the resulting blended data encodedinto a luminance component of resulting combined images.

For example, any of the techniques disclosed in the followingapplications may be used in various embodiments: U.S. patent applicationSer. No. 12/477,828 filed Jun. 3, 2009; U.S. patent application Ser. No.12/766,739 filed Apr. 23, 2010; U.S. patent application Ser. No.13/105,765 filed May 11, 2011; U.S. patent application Ser. No.13/437,645 filed Apr. 2, 2012; U.S. Provisional Patent Application No.61/473,207 filed Apr. 8, 2011; U.S. Provisional Patent Application No.61/746,069 filed Dec. 26, 2012; U.S. Provisional Patent Application No.61/746,074 filed Dec. 26, 2012; U.S. Provisional Patent Application No.61/748,018 filed Dec. 31, 2012; U.S. Provisional Patent Application No.61/792,582 filed Mar. 15, 2013; U.S. Provisional Patent Application No.61/793,952 filed Mar. 15, 2013; and International Patent Application No.PCT/EP2011/056432 filed Apr. 21, 2011, all of such applications areincorporated herein by reference in their entirety. Any of thetechniques described herein, or described in other applications orpatents referenced herein, may be applied to any of the various thermaldevices, non-thermal devices, and uses described herein.

Referring again to FIG. 12, in various embodiments, host device 2102 mayinclude shutter 2105. In this regard, shutter 2105 may be selectivelypositioned over socket 2104 (e.g., as identified by arrows 2103) whileinfrared imaging module 2100 is installed therein. In this regard,shutter 2105 may be used, for example, to protect infrared imagingmodule 2100 when not in use. Shutter 2105 may also be used as atemperature reference as part of a calibration process (e.g., a NUCprocess or other calibration processes) for infrared imaging module 2100as would be understood by one skilled in the art.

In various embodiments, shutter 2105 may be made from various materialssuch as, for example, polymers, glass, aluminum (e.g., painted oranodized) or other materials. In various embodiments, shutter 2105 mayinclude one or more coatings to selectively filter electromagneticradiation and/or adjust various optical properties of shutter 2105(e.g., a uniform blackbody coating or a reflective gold coating).

In another embodiment, shutter 2105 may be fixed in place to protectinfrared imaging module 2100 at all times. In this case, shutter 2105 ora portion of shutter 2105 may be made from appropriate materials (e.g.,polymers or infrared transmitting materials such as silicon, germanium,zinc selenide, or chalcogenide glasses) that do not substantially filterdesired infrared wavelengths. In another embodiment, a shutter may beimplemented as part of infrared imaging module 2100 (e.g., within or aspart of a lens barrel or other components of infrared imaging module2100), as would be understood by one skilled in the art.

Alternatively, in another embodiment, a shutter (e.g., shutter 2105 orother type of external or internal shutter) need not be provided, butrather a NUC process or other type of calibration may be performed usingshutterless techniques. In another embodiment, a NUC process or othertype of calibration using shutterless techniques may be performed incombination with shutter-based techniques.

Infrared imaging module 2100 and host device 2102 may be implemented inaccordance with any of the various techniques set forth in U.S.Provisional Patent Application No. 61/495,873 filed Jun. 10, 2011, U.S.Provisional Patent Application No. 61/495,879 filed Jun. 10, 2011, andU.S. Provisional Patent Application No. 61/495,888 filed Jun. 10, 2011,which are incorporated herein by reference in their entirety.

In various embodiments, the components of host device 2102 and/orinfrared imaging module 2100 may be implemented as a local ordistributed system with components in communication with each other overwired and/or wireless networks. Accordingly, the various operationsidentified in this disclosure may be performed by local and/or remotecomponents as may be desired in particular implementations.

FIG. 16 illustrates a flow diagram of various operations to determineNUC terms in accordance with an embodiment of the disclosure. In someembodiments, the operations of FIG. 16 may be performed by processingmodule 2160 or processor 2195 (both also generally referred to as aprocessor) operating on image frames captured by infrared sensors 2132.

In block 2505, infrared sensors 2132 begin capturing image frames of ascene. Typically, the scene will be the real world environment in whichhost device 2102 is currently located. In this regard, shutter 2105 (ifoptionally provided) may be opened to permit infrared imaging module toreceive infrared radiation from the scene. Infrared sensors 2132 maycontinue capturing image frames during all operations shown in FIG. 16.In this regard, the continuously captured image frames may be used forvarious operations as further discussed. In one embodiment, the capturedimage frames may be temporally filtered (e.g., in accordance with theprocess of block 2826 further described herein with regard to FIG. 19)and be processed by other terms (e.g., factory gain terms 2812, factoryoffset terms 2816, previously determined NUC terms 2817, column FPNterms 2820, and row FPN terms 2824 as further described herein withregard to FIG. 19) before they are used in the operations shown in FIG.16.

In block 2510, a NUC process initiating event is detected. In oneembodiment, the NUC process may be initiated in response to physicalmovement of host device 2102. Such movement may be detected, forexample, by motion sensors 2194 which may be polled by a processor. Inone example, a user may move host device 2102 in a particular manner,such as by intentionally waving host device 2102 back and forth in an“erase” or “swipe” movement. In this regard, the user may move hostdevice 2102 in accordance with a predetermined speed and direction(velocity), such as in an up and down, side to side, or other pattern toinitiate the NUC process. In this example, the use of such movements maypermit the user to intuitively operate host device 2102 to simulate the“erasing” of noise in captured image frames.

In another example, a NUC process may be initiated by host device 2102if motion exceeding a threshold value is detected (e.g., motion greaterthan expected for ordinary use). It is contemplated that any desiredtype of spatial translation of host device 2102 may be used to initiatethe NUC process.

In yet another example, a NUC process may be initiated by host device2102 if a minimum time has elapsed since a previously performed NUCprocess. In a further example, a NUC process may be initiated by hostdevice 2102 if infrared imaging module 2100 has experienced a minimumtemperature change since a previously performed NUC process. In a stillfurther example, a NUC process may be continuously initiated andrepeated.

In block 2515, after a NUC process initiating event is detected, it isdetermined whether the NUC process should actually be performed. In thisregard, the NUC process may be selectively initiated based on whetherone or more additional conditions are met. For example, in oneembodiment, the NUC process may not be performed unless a minimum timehas elapsed since a previously performed NUC process. In anotherembodiment, the NUC process may not be performed unless infrared imagingmodule 2100 has experienced a minimum temperature change since apreviously performed NUC process. Other criteria or conditions may beused in other embodiments. If appropriate criteria or conditions havebeen met, then the flow diagram continues to block 2520. Otherwise, theflow diagram returns to block 2505.

In the NUC process, blurred image frames may be used to determine NUCterms which may be applied to captured image frames to correct for FPN.As discussed, in one embodiment, the blurred image frames may beobtained by accumulating multiple image frames of a moving scene (e.g.,captured while the scene and/or the thermal imager is in motion). Inanother embodiment, the blurred image frames may be obtained bydefocusing an optical element or other component of the thermal imager.

Accordingly, in block 2520 a choice of either approach is provided. Ifthe motion-based approach is used, then the flow diagram continues toblock 2525. If the defocus-based approach is used, then the flow diagramcontinues to block 2530.

Referring now to the motion-based approach, in block 2525 motion isdetected. For example, in one embodiment, motion may be detected basedon the image frames captured by infrared sensors 2132. In this regard,an appropriate motion detection process (e.g., an image registrationprocess, a frame-to-frame difference calculation, or other appropriateprocess) may be applied to captured image frames to determine whethermotion is present (e.g., whether static or moving image frames have beencaptured). For example, in one embodiment, it can be determined whetherpixels or regions around the pixels of consecutive image frames havechanged more than a user defined amount (e.g., a percentage and/orthreshold value). If at least a given percentage of pixels have changedby at least the user defined amount, then motion will be detected withsufficient certainty to proceed to block 2535.

In another embodiment, motion may be determined on a per pixel basis,wherein only pixels that exhibit significant changes are accumulated toprovide the blurred image frame. For example, counters may be providedfor each pixel and used to ensure that the same number of pixel valuesare accumulated for each pixel, or used to average the pixel valuesbased on the number of pixel values actually accumulated for each pixel.Other types of image-based motion detection may be performed such asperforming a Radon transform.

In another embodiment, motion may be detected based on data provided bymotion sensors 2194. In one embodiment, such motion detection mayinclude detecting whether host device 2102 is moving along a relativelystraight trajectory through space. For example, if host device 2102 ismoving along a relatively straight trajectory, then it is possible thatcertain objects appearing in the imaged scene may not be sufficientlyblurred (e.g., objects in the scene that may be aligned with or movingsubstantially parallel to the straight trajectory). Thus, in such anembodiment, the motion detected by motion sensors 2194 may beconditioned on host device 2102 exhibiting, or not exhibiting,particular trajectories.

In yet another embodiment, both a motion detection process and motionsensors 2194 may be used. Thus, using any of these various embodiments,a determination can be made as to whether or not each image frame wascaptured while at least a portion of the scene and host device 2102 werein motion relative to each other (e.g., which may be caused by hostdevice 2102 moving relative to the scene, at least a portion of thescene moving relative to host device 2102, or both).

It is expected that the image frames for which motion was detected mayexhibit some secondary blurring of the captured scene (e.g., blurredthermal image data associated with the scene) due to the thermal timeconstants of infrared sensors 2132 (e.g., microbolometer thermal timeconstants) interacting with the scene movement.

In block 2535, image frames for which motion was detected areaccumulated. For example, if motion is detected for a continuous seriesof image frames, then the image frames of the series may be accumulated.As another example, if motion is detected for only some image frames,then the non-moving image frames may be skipped and not included in theaccumulation. Thus, a continuous or discontinuous set of image framesmay be selected to be accumulated based on the detected motion.

In block 2540, the accumulated image frames are averaged to provide ablurred image frame. Because the accumulated image frames were capturedduring motion, it is expected that actual scene information will varybetween the image frames and thus cause the scene information to befurther blurred in the resulting blurred image frame (block 2545).

In contrast, FPN (e.g., caused by one or more components of infraredimaging module 2100) will remain fixed over at least short periods oftime and over at least limited changes in scene irradiance duringmotion. As a result, image frames captured in close proximity in timeand space during motion will suffer from identical or at least verysimilar FPN. Thus, although scene information may change in consecutiveimage frames, the FPN will stay essentially constant. By averaging,multiple image frames captured during motion will blur the sceneinformation, but will not blur the FPN. As a result, FPN will remainmore clearly defined in the blurred image frame provided in block 2545than the scene information.

In one embodiment, 32 or more image frames are accumulated and averagedin blocks 2535 and 2540. However, any desired number of image frames maybe used in other embodiments, but with generally decreasing correctionaccuracy as frame count is decreased.

Referring now to the defocus-based approach, in block 2530, a defocusoperation may be performed to intentionally defocus the image framescaptured by infrared sensors 2132. For example, in one embodiment, oneor more actuators 2199 may be used to adjust, move, or otherwisetranslate optical element 2180, infrared sensor assembly 2128, and/orother components of infrared imaging module 2100 to cause infraredsensors 2132 to capture a blurred (e.g., unfocused) image frame of thescene. Other non-actuator based techniques are also contemplated forintentionally defocusing infrared image frames such as, for example,manual (e.g., user-initiated) defocusing.

Although the scene may appear blurred in the image frame, FPN (e.g.,caused by one or more components of infrared imaging module 2100) willremain unaffected by the defocusing operation. As a result, a blurredimage frame of the scene will be provided (block 2545) with FPNremaining more clearly defined in the blurred image than the sceneinformation.

In the above discussion, the defocus-based approach has been describedwith regard to a single captured image frame. In another embodiment, thedefocus-based approach may include accumulating multiple image frameswhile the infrared imaging module 2100 has been defocused and averagingthe defocused image frames to remove the effects of temporal noise andprovide a blurred image frame in block 2545.

Thus, it will be appreciated that a blurred image frame may be providedin block 2545 by either the motion-based approach or the defocus-basedapproach. Because much of the scene information will be blurred byeither motion, defocusing, or both, the blurred image frame may beeffectively considered a low pass filtered version of the originalcaptured image frames with respect to scene information.

In block 2550, the blurred image frame is processed to determine updatedrow and column FPN terms (e.g., if row and column FPN terms have notbeen previously determined then the updated row and column FPN terms maybe new row and column FPN terms in the first iteration of block 2550).As used in this disclosure, the terms row and column may be usedinterchangeably depending on the orientation of infrared sensors 2132and/or other components of infrared imaging module 2100.

In one embodiment, block 2550 includes determining a spatial FPNcorrection term for each row of the blurred image frame (e.g., each rowmay have its own spatial FPN correction term), and also determining aspatial FPN correction term for each column of the blurred image frame(e.g., each column may have its own spatial FPN correction term). Suchprocessing may be used to reduce the spatial and slowly varying (1/f)row and column FPN inherent in thermal imagers caused by, for example,1/f noise characteristics of amplifiers in ROIC 2402 which may manifestas vertical and horizontal stripes in image frames.

Advantageously, by determining spatial row and column FPN terms usingthe blurred image frame, there will be a reduced risk of vertical andhorizontal objects in the actual imaged scene from being mistaken forrow and column noise (e.g., real scene content will be blurred while FPNremains unblurred).

In one embodiment, row and column FPN terms may be determined byconsidering differences between neighboring pixels of the blurred imageframe. For example, FIG. 17 illustrates differences between neighboringpixels in accordance with an embodiment of the disclosure. Specifically,in FIG. 17 a pixel 2610 is compared to its 8 nearest horizontalneighbors: d0-d3 on one side and d4-d7 on the other side. Differencesbetween the neighbor pixels can be averaged to obtain an estimate of theoffset error of the illustrated group of pixels. An offset error may becalculated for each pixel in a row or column and the average result maybe used to correct the entire row or column.

To prevent real scene data from being interpreted as noise, upper andlower threshold values may be used (thPix and −thPix). Pixel valuesfalling outside these threshold values (pixels d1 and d4 in thisexample) are not used to obtain the offset error. In addition, themaximum amount of row and column FPN correction may be limited by thesethreshold values.

Further techniques for performing spatial row and column FPN correctionprocessing are set forth in U.S. patent application Ser. No. 12/396,340filed Mar. 2, 2009 which is incorporated herein by reference in itsentirety.

Referring again to FIG. 16, the updated row and column FPN termsdetermined in block 2550 are stored (block 2552) and applied (block2555) to the blurred image frame provided in block 2545. After theseterms are applied, some of the spatial row and column FPN in the blurredimage frame may be reduced. However, because such terms are appliedgenerally to rows and columns, additional FPN may remain such asspatially uncorrelated FPN associated with pixel to pixel drift or othercauses. Neighborhoods of spatially correlated FPN may also remain whichmay not be directly associated with individual rows and columns.Accordingly, further processing may be performed as discussed below todetermine NUC terms.

In block 2560, local contrast values (e.g., edges or absolute values ofgradients between adjacent or small groups of pixels) in the blurredimage frame are determined. If scene information in the blurred imageframe includes contrasting areas that have not been significantlyblurred (e.g., high contrast edges in the original scene data), thensuch features may be identified by a contrast determination process inblock 2560.

For example, local contrast values in the blurred image frame may becalculated, or any other desired type of edge detection process may beapplied to identify certain pixels in the blurred image as being part ofan area of local contrast. Pixels that are marked in this manner may beconsidered as containing excessive high spatial frequency sceneinformation that would be interpreted as FPN (e.g., such regions maycorrespond to portions of the scene that have not been sufficientlyblurred). As such, these pixels may be excluded from being used in thefurther determination of NUC terms. In one embodiment, such contrastdetection processing may rely on a threshold that is higher than theexpected contrast value associated with FPN (e.g., pixels exhibiting acontrast value higher than the threshold may be considered to be sceneinformation, and those lower than the threshold may be considered to beexhibiting FPN).

In one embodiment, the contrast determination of block 2560 may beperformed on the blurred image frame after row and column FPN terms havebeen applied to the blurred image frame (e.g., as shown in FIG. 16). Inanother embodiment, block 2560 may be performed prior to block 2550 todetermine contrast before row and column FPN terms are determined (e.g.,to prevent scene based contrast from contributing to the determinationof such terms).

Following block 2560, it is expected that any high spatial frequencycontent remaining in the blurred image frame may be generally attributedto spatially uncorrelated FPN. In this regard, following block 2560,much of the other noise or actual desired scene based information hasbeen removed or excluded from the blurred image frame due to:intentional blurring of the image frame (e.g., by motion or defocusingin blocks 2520 through 2545), application of row and column FPN terms(block 2555), and contrast determination (block 2560).

Thus, it can be expected that following block 2560, any remaining highspatial frequency content (e.g., exhibited as areas of contrast ordifferences in the blurred image frame) may be attributed to spatiallyuncorrelated FPN. Accordingly, in block 2565, the blurred image frame ishigh pass filtered. In one embodiment, this may include applying a highpass filter to extract the high spatial frequency content from theblurred image frame. In another embodiment, this may include applying alow pass filter to the blurred image frame and taking a differencebetween the low pass filtered image frame and the unfiltered blurredimage frame to obtain the high spatial frequency content. In accordancewith various embodiments of the present disclosure, a high pass filtermay be implemented by calculating a mean difference between a sensorsignal (e.g., a pixel value) and its neighbors.

In block 2570, a flat field correction process is performed on the highpass filtered blurred image frame to determine updated NUC terms (e.g.,if a NUC process has not previously been performed then the updated NUCterms may be new NUC terms in the first iteration of block 2570).

For example, FIG. 18 illustrates a flat field correction technique 2700in accordance with an embodiment of the disclosure. In FIG. 18, a NUCterm may be determined for each pixel 2710 of the blurred image frameusing the values of its neighboring pixels 2712 to 2726. For each pixel2710, several gradients may be determined based on the absolutedifference between the values of various adjacent pixels. For example,absolute value differences may be determined between: pixels 2712 and2714 (a left to right diagonal gradient), pixels 2716 and 2718 (a top tobottom vertical gradient), pixels 2720 and 2722 (a right to leftdiagonal gradient), and pixels 2724 and 2726 (a left to right horizontalgradient).

These absolute differences may be summed to provide a summed gradientfor pixel 2710. A weight value may be determined for pixel 2710 that isinversely proportional to the summed gradient. This process may beperformed for all pixels 2710 of the blurred image frame until a weightvalue is provided for each pixel 2710. For areas with low gradients(e.g., areas that are blurry or have low contrast), the weight valuewill be close to one. Conversely, for areas with high gradients, theweight value will be zero or close to zero. The update to the NUC termas estimated by the high pass filter is multiplied with the weightvalue.

In one embodiment, the risk of introducing scene information into theNUC terms can be further reduced by applying some amount of temporaldamping to the NUC term determination process. For example, a temporaldamping factor X, between 0 and 1 may be chosen such that the new NUCterm (NUC_(NEW)) stored is a weighted average of the old NUC term(NUC_(OLD)) and the estimated updated NUC term (NUC_(UPDATE)). In oneembodiment, this can be expressed asNUC_(NEW)=λ·NUC_(OLD)+(1−λ)·(NUC_(OLD)+NUC_(UPDATE)).

Although the determination of NUC terms has been described with regardto gradients, local contrast values may be used instead whereappropriate. Other techniques may also be used such as, for example,standard deviation calculations. Other types flat field correctionprocesses may be performed to determine NUC terms including, forexample, various processes identified in U.S. Pat. No. 6,028,309 issuedFeb. 22, 2000, U.S. Pat. No. 6,812,465 issued Nov. 2, 2004, and U.S.patent application Ser. No. 12/114,865 filed May 5, 2008, which areincorporated herein by reference in their entirety.

Referring again to FIG. 16, block 2570 may include additional processingof the NUC terms. For example, in one embodiment, to preserve the scenesignal mean, the sum of all NUC terms may be normalized to zero bysubtracting the NUC term mean from each NUC term. Also in block 2570, toavoid row and column noise from affecting the NUC terms, the mean valueof each row and column may be subtracted from the NUC terms for each rowand column. As a result, row and column FPN filters using the row andcolumn FPN terms determined in block 2550 may be better able to filterout row and column noise in further iterations (e.g., as further shownin FIG. 19) after the NUC terms are applied to captured images (e.g., inblock 2580 further discussed herein). In this regard, the row and columnFPN filters may in general use more data to calculate the per row andper column offset coefficients (e.g., row and column FPN terms) and maythus provide a more robust alternative for reducing spatially correlatedFPN than the NUC terms which are based on high pass filtering to capturespatially uncorrelated noise.

In blocks 2571-2573, additional high pass filtering and furtherdeterminations of updated NUC terms may be optionally performed toremove spatially correlated FPN with lower spatial frequency thanpreviously removed by row and column FPN terms. In this regard, somevariability in infrared sensors 2132 or other components of infraredimaging module 2100 may result in spatially correlated FPN noise thatcannot be easily modeled as row or column noise. Such spatiallycorrelated FPN may include, for example, window defects on a sensorpackage or a cluster of infrared sensors 2132 that respond differentlyto irradiance than neighboring infrared sensors 2132. In one embodiment,such spatially correlated FPN may be mitigated with an offsetcorrection. If the amount of such spatially correlated FPN issignificant, then the noise may also be detectable in the blurred imageframe. Since this type of noise may affect a neighborhood of pixels, ahigh pass filter with a small kernel may not detect the FPN in theneighborhood (e.g., all values used in high pass filter may be takenfrom the neighborhood of affected pixels and thus may be affected by thesame offset error). For example, if the high pass filtering of block2565 is performed with a small kernel (e.g., considering onlyimmediately adjacent pixels that fall within a neighborhood of pixelsaffected by spatially correlated FPN), then broadly distributedspatially correlated FPN may not be detected.

For example, FIG. 22 illustrates spatially correlated FPN in aneighborhood of pixels in accordance with an embodiment of thedisclosure. As shown in a sample image frame 21100, a neighborhood ofpixels 21110 may exhibit spatially correlated FPN that is not preciselycorrelated to individual rows and columns and is distributed over aneighborhood of several pixels (e.g., a neighborhood of approximately 4by 4 pixels in this example). Sample image frame 21100 also includes aset of pixels 21120 exhibiting substantially uniform response that arenot used in filtering calculations, and a set of pixels 21130 that areused to estimate a low pass value for the neighborhood of pixels 21110.In one embodiment, pixels 21130 may be a number of pixels divisible bytwo in order to facilitate efficient hardware or software calculations.

Referring again to FIG. 16, in blocks 2571-2573, additional high passfiltering and further determinations of updated NUC terms may beoptionally performed to remove spatially correlated FPN such asexhibited by pixels 21110. In block 2571, the updated NUC termsdetermined in block 2570 are applied to the blurred image frame. Thus,at this time, the blurred image frame will have been initially correctedfor spatially correlated FPN (e.g., by application of the updated rowand column FPN terms in block 2555), and also initially corrected forspatially uncorrelated FPN (e.g., by application of the updated NUCterms applied in block 2571).

In block 2572, a further high pass filter is applied with a largerkernel than was used in block 2565, and further updated NUC terms may bedetermined in block 2573. For example, to detect the spatiallycorrelated FPN present in pixels 21110, the high pass filter applied inblock 2572 may include data from a sufficiently large enoughneighborhood of pixels such that differences can be determined betweenunaffected pixels (e.g., pixels 21120) and affected pixels (e.g., pixels21110). For example, a low pass filter with a large kernel can be used(e.g., an N by N kernel that is much greater than 3 by 3 pixels) and theresults may be subtracted to perform appropriate high pass filtering.

In one embodiment, for computational efficiency, a sparse kernel may beused such that only a small number of neighboring pixels inside an N byN neighborhood are used. For any given high pass filter operation usingdistant neighbors (e.g., a large kernel), there is a risk of modelingactual (potentially blurred) scene information as spatially correlatedFPN. Accordingly, in one embodiment, the temporal damping factor λ maybe set close to 1 for updated NUC terms determined in block 2573.

In various embodiments, blocks 2571-2573 may be repeated (e.g.,cascaded) to iteratively perform high pass filtering with increasingkernel sizes to provide further updated NUC terms further correct forspatially correlated FPN of desired neighborhood sizes. In oneembodiment, the decision to perform such iterations may be determined bywhether spatially correlated FPN has actually been removed by theupdated NUC terms of the previous performance of blocks 2571-2573.

After blocks 2571-2573 are finished, a decision is made regardingwhether to apply the updated NUC terms to captured image frames (block2574). For example, if an average of the absolute value of the NUC termsfor the entire image frame is less than a minimum threshold value, orgreater than a maximum threshold value, the NUC terms may be deemedspurious or unlikely to provide meaningful correction. Alternatively,thresholding criteria may be applied to individual pixels to determinewhich pixels receive updated NUC terms. In one embodiment, the thresholdvalues may correspond to differences between the newly calculated NUCterms and previously calculated NUC terms. In another embodiment, thethreshold values may be independent of previously calculated NUC terms.Other tests may be applied (e.g., spatial correlation tests) todetermine whether the NUC terms should be applied.

If the NUC terms are deemed spurious or unlikely to provide meaningfulcorrection, then the flow diagram returns to block 2505. Otherwise, thenewly determined NUC terms are stored (block 2575) to replace previousNUC terms (e.g., determined by a previously performed iteration of FIG.16) and applied (block 2580) to captured image frames.

FIG. 19 illustrates various image processing techniques of FIG. 16 andother operations applied in an image processing pipeline 2800 inaccordance with an embodiment of the disclosure. In this regard,pipeline 2800 identifies various operations of FIG. 16 in the context ofan overall iterative image processing scheme for correcting image framesprovided by infrared imaging module 2100. In some embodiments, pipeline2800 may be provided by processing module 2160 or processor 2195 (bothalso generally referred to as a processor) operating on image framescaptured by infrared sensors 2132.

Image frames captured by infrared sensors 2132 may be provided to aframe averager 2804 that integrates multiple image frames to provideimage frames 2802 with an improved signal to noise ratio. Frame averager2804 may be effectively provided by infrared sensors 2132, ROIC 2402,and other components of infrared sensor assembly 2128 that areimplemented to support high image capture rates. For example, in oneembodiment, infrared sensor assembly 2128 may capture infrared imageframes at a frame rate of 240 Hz (e.g., 240 images per second). In thisembodiment, such a high frame rate may be implemented, for example, byoperating infrared sensor assembly 2128 at relatively low voltages(e.g., compatible with mobile telephone voltages) and by using arelatively small array of infrared sensors 2132 (e.g., an array of 64 by64 infrared sensors in one embodiment).

In one embodiment, such infrared image frames may be provided frominfrared sensor assembly 2128 to processing module 2160 at a high framerate (e.g., 240 Hz or other frame rates). In another embodiment,infrared sensor assembly 2128 may integrate over longer time periods, ormultiple time periods, to provide integrated (e.g., averaged) infraredimage frames to processing module 2160 at a lower frame rate (e.g., 30Hz, 9 Hz, or other frame rates). Further information regardingimplementations that may be used to provide high image capture rates maybe found in U.S. Provisional Patent Application No. 61/495,879 filedJun. 10, 2011 which is incorporated herein by reference in its entirety.

Image frames 2802 proceed through pipeline 2800 where they are adjustedby various terms, temporally filtered, used to determine the variousadjustment terms, and gain compensated.

In blocks 2810 and 2814, factory gain terms 2812 and factory offsetterms 2816 are applied to image frames 2802 to compensate for gain andoffset differences, respectively, between the various infrared sensors2132 and/or other components of infrared imaging module 2100 determinedduring manufacturing and testing.

In block 2580, NUC terms 2817 are applied to image frames 2802 tocorrect for FPN as discussed. In one embodiment, if NUC terms 2817 havenot yet been determined (e.g., before a NUC process has been initiated),then block 2580 may not be performed or initialization values may beused for NUC terms 2817 that result in no alteration to the image data(e.g., offsets for every pixel would be equal to zero).

In blocks 2818 and 2822, column FPN terms 2820 and row FPN terms 2824,respectively, are applied to image frames 2802. Column FPN terms 2820and row FPN terms 2824 may be determined in accordance with block 2550as discussed. In one embodiment, if the column FPN terms 2820 and rowFPN terms 2824 have not yet been determined (e.g., before a NUC processhas been initiated), then blocks 2818 and 2822 may not be performed orinitialization values may be used for the column FPN terms 2820 and rowFPN terms 2824 that result in no alteration to the image data (e.g.,offsets for every pixel would be equal to zero).

In block 2826, temporal filtering is performed on image frames 2802 inaccordance with a temporal noise reduction (TNR) process. FIG. 20illustrates a TNR process in accordance with an embodiment of thedisclosure. In FIG. 20, a presently received image frame 2802 a and apreviously temporally filtered image frame 2802 b are processed todetermine a new temporally filtered image frame 2802 e. Image frames2802 a and 2802 b include local neighborhoods of pixels 2803 a and 2803b centered around pixels 2805 a and 2805 b, respectively. Neighborhoods2803 a and 2803 b correspond to the same locations within image frames2802 a and 2802 b and are subsets of the total pixels in image frames2802 a and 2802 b. In the illustrated embodiment, neighborhoods 2803 aand 2803 b include areas of 5 by 5 pixels. Other neighborhood sizes maybe used in other embodiments.

Differences between corresponding pixels of neighborhoods 2803 a and2803 b are determined and averaged to provide an averaged delta value2805 c for the location corresponding to pixels 2805 a and 2805 b.Averaged delta value 2805 c may be used to determine weight values inblock 2807 to be applied to pixels 2805 a and 2805 b of image frames2802 a and 2802 b.

In one embodiment, as shown in graph 2809, the weight values determinedin block 2807 may be inversely proportional to averaged delta value 2805c such that weight values drop rapidly towards zero when there are largedifferences between neighborhoods 2803 a and 2803 b. In this regard,large differences between neighborhoods 2803 a and 2803 b may indicatethat changes have occurred within the scene (e.g., due to motion) andpixels 2802 a and 2802 b may be appropriately weighted, in oneembodiment, to avoid introducing blur across frame-to-frame scenechanges. Other associations between weight values and averaged deltavalue 2805 c may be used in various embodiments.

The weight values determined in block 2807 may be applied to pixels 2805a and 2805 b to determine a value for corresponding pixel 2805 e ofimage frame 2802 e (block 2811). In this regard, pixel 2805 e may have avalue that is a weighted average (or other combination) of pixels 2805 aand 2805 b, depending on averaged delta value 2805 c and the weightvalues determined in block 2807.

For example, pixel 2805 e of temporally filtered image frame 2802 e maybe a weighted sum of pixels 2805 a and 2805 b of image frames 2802 a and2802 b. If the average difference between pixels 2805 a and 2805 b isdue to noise, then it may be expected that the average change betweenneighborhoods 2805 a and 2805 b will be close to zero (e.g.,corresponding to the average of uncorrelated changes). Under suchcircumstances, it may be expected that the sum of the differencesbetween neighborhoods 2805 a and 2805 b will be close to zero. In thiscase, pixel 2805 a of image frame 2802 a may both be appropriatelyweighted so as to contribute to the value of pixel 2805 e.

However, if the sum of such differences is not zero (e.g., evendiffering from zero by a small amount in one embodiment), then thechanges may be interpreted as being attributed to motion instead ofnoise. Thus, motion may be detected based on the average changeexhibited by neighborhoods 2805 a and 2805 b. Under these circumstances,pixel 2805 a of image frame 2802 a may be weighted heavily, while pixel2805 b of image frame 2802 b may be weighted lightly.

Other embodiments are also contemplated. For example, although averageddelta value 2805 c has been described as being determined based onneighborhoods 2805 a and 2805 b, in other embodiments averaged deltavalue 2805 c may be determined based on any desired criteria (e.g.,based on individual pixels or other types of groups of sets of pixels).

In the above embodiments, image frame 2802 a has been described as apresently received image frame and image frame 2802 b has been describedas a previously temporally filtered image frame. In another embodiment,image frames 2802 a and 2802 b may be first and second image framescaptured by infrared imaging module 2100 that have not been temporallyfiltered.

FIG. 21 illustrates further implementation details in relation to theTNR process of block 2826. As shown in FIG. 21, image frames 2802 a and2802 b may be read into line buffers 21010 a and 21010 b, respectively,and image frame 2802 b (e.g., the previous image frame) may be stored ina frame buffer 21020 before being read into line buffer 21010 b. In oneembodiment, line buffers 21010 a-b and frame buffer 21020 may beimplemented by a block of random access memory (RAM) provided by anyappropriate component of infrared imaging module 2100 and/or host device2102.

Referring again to FIG. 19, image frame 2802 e may be passed to anautomatic gain compensation block 2828 for further processing to providea result image frame 2830 that may be used by host device 2102 asdesired.

FIG. 19 further illustrates various operations that may be performed todetermine row and column FPN terms and NUC terms as discussed. In oneembodiment, these operations may use image frames 2802 e as shown inFIG. 19. Because image frames 2802 e have already been temporallyfiltered, at least some temporal noise may be removed and thus will notinadvertently affect the determination of row and column FPN terms 2824and 2820 and NUC terms 2817. In another embodiment, non-temporallyfiltered image frames 2802 may be used.

In FIG. 19, blocks 2510, 2515, and 2520 of FIG. 16 are collectivelyrepresented together. As discussed, a NUC process may be selectivelyinitiated and performed in response to various NUC process initiatingevents and based on various criteria or conditions. As also discussed,the NUC process may be performed in accordance with a motion-basedapproach (blocks 2525, 2535, and 2540) or a defocus-based approach(block 2530) to provide a blurred image frame (block 2545). FIG. 19further illustrates various additional blocks 2550, 2552, 2555, 2560,2565, 2570, 2571, 2572, 2573, and 2575 previously discussed with regardto FIG. 16.

As shown in FIG. 19, row and column FPN terms 2824 and 2820 and NUCterms 2817 may be determined and applied in an iterative fashion suchthat updated terms are determined using image frames 2802 to whichprevious terms have already been applied. As a result, the overallprocess of FIG. 19 may repeatedly update and apply such terms tocontinuously reduce the noise in image frames 2830 to be used by hostdevice 2102.

Referring again to FIG. 21, further implementation details areillustrated for various blocks of FIGS. 5 and 8 in relation to pipeline2800. For example, blocks 2525, 2535, and 2540 are shown as operating atthe normal frame rate of image frames 2802 received by pipeline 2800. Inthe embodiment shown in FIG. 21, the determination made in block 2525 isrepresented as a decision diamond used to determine whether a givenimage frame 2802 has sufficiently changed such that it may be consideredan image frame that will enhance the blur if added to other image framesand is therefore accumulated (block 2535 is represented by an arrow inthis embodiment) and averaged (block 2540).

Also in FIG. 21, the determination of column FPN terms 2820 (block 2550)is shown as operating at an update rate that in this example is 1/32 ofthe sensor frame rate (e.g., normal frame rate) due to the averagingperformed in block 2540. Other update rates may be used in otherembodiments. Although only column FPN terms 2820 are identified in FIG.21, row FPN terms 2824 may be implemented in a similar fashion at thereduced frame rate.

FIG. 21 also illustrates further implementation details in relation tothe NUC determination process of block 2570. In this regard, the blurredimage frame may be read to a line buffer 21030 (e.g., implemented by ablock of RAM provided by any appropriate component of infrared imagingmodule 2100 and/or host device 2102). The flat field correctiontechnique 2700 of FIG. 18 may be performed on the blurred image frame.

In view of the present disclosure, it will be appreciated thattechniques described herein may be used to remove various types of FPN(e.g., including very high amplitude FPN) such as spatially correlatedrow and column FPN and spatially uncorrelated FPN.

Other embodiments are also contemplated. For example, in one embodiment,the rate at which row and column FPN terms and/or NUC terms are updatedcan be inversely proportional to the estimated amount of blur in theblurred image frame and/or inversely proportional to the magnitude oflocal contrast values (e.g., determined in block 2560).

In various embodiments, the described techniques may provide advantagesover conventional shutter-based noise correction techniques. Forexample, by using a shutterless process, a shutter (e.g., such asshutter 2105) need not be provided, thus permitting reductions in size,weight, cost, and mechanical complexity. Power and maximum voltagesupplied to, or generated by, infrared imaging module 2100 may also bereduced if a shutter does not need to be mechanically operated.Reliability will be improved by removing the shutter as a potentialpoint of failure. A shutterless process also eliminates potential imageinterruption caused by the temporary blockage of the imaged scene by ashutter.

Also, by correcting for noise using intentionally blurred image framescaptured from a real world scene (not a uniform scene provided by ashutter), noise correction may be performed on image frames that haveirradiance levels similar to those of the actual scene desired to beimaged. This can improve the accuracy and effectiveness of noisecorrection terms determined in accordance with the various describedtechniques.

As discussed, in various embodiments, infrared imaging module 2100 maybe configured to operate at low voltage levels. In particular, infraredimaging module 2100 may be implemented with circuitry configured tooperate at low power and/or in accordance with other parameters thatpermit infrared imaging module 2100 to be conveniently and effectivelyimplemented in various types of host devices 2102, such as mobiledevices and other devices.

For example, FIG. 23 illustrates a block diagram of anotherimplementation of infrared sensor assembly 2128 including infraredsensors 2132 and an LDO 21220 in accordance with an embodiment of thedisclosure. As shown, FIG. 23 also illustrates various components 21202,21204, 21205, 21206, 21208, and 21210 which may implemented in the sameor similar manner as corresponding components previously described withregard to FIG. 15. FIG. 23 also illustrates bias correction circuitry21212 which may be used to adjust one or more bias voltages provided toinfrared sensors 2132 (e.g., to compensate for temperature changes,self-heating, and/or other factors).

In some embodiments, LDO 21220 may be provided as part of infraredsensor assembly 2128 (e.g., on the same chip and/or wafer level packageas the ROIC). For example, LDO 21220 may be provided as part of an FPAwith infrared sensor assembly 2128. As discussed, such implementationsmay reduce power supply noise introduced to infrared sensor assembly2128 and thus provide an improved PSRR. In addition, by implementing theLDO with the ROIC, less die area may be consumed and fewer discrete die(or chips) are needed.

LDO 21220 receives an input voltage provided by a power source 21230over a supply line 21232. LDO 21220 provides an output voltage tovarious components of infrared sensor assembly 2128 over supply lines21222. In this regard, LDO 21220 may provide substantially identicalregulated output voltages to various components of infrared sensorassembly 2128 in response to a single input voltage received from powersource 21230, in accordance with various techniques described in, forexample, U.S. patent application Ser. No. 14/101,245 filed Dec. 9, 2013incorporated herein by reference in its entirety.

For example, in some embodiments, power source 21230 may provide aninput voltage in a range of approximately 2.8 volts to approximately 11volts (e.g., approximately 2.8 volts in one embodiment), and LDO 21220may provide an output voltage in a range of approximately 1.5 volts toapproximately 2.8 volts (e.g., approximately 2.8, 2.5, 2.4, and/or lowervoltages in various embodiments). In this regard, LDO 21220 may be usedto provide a consistent regulated output voltage, regardless of whetherpower source 21230 is implemented with a conventional voltage range ofapproximately 9 volts to approximately 11 volts, or a low voltage suchas approximately 2.8 volts. As such, although various voltage ranges areprovided for the input and output voltages, it is contemplated that theoutput voltage of LDO 21220 will remain fixed despite changes in theinput voltage.

The implementation of LDO 21220 as part of infrared sensor assembly 2128provides various advantages over conventional power implementations forFPAs. For example, conventional FPAs typically rely on multiple powersources, each of which may be provided separately to the FPA, andseparately distributed to the various components of the FPA. Byregulating a single power source 21230 by LDO 21220, appropriatevoltages may be separately provided (e.g., to reduce possible noise) toall components of infrared sensor assembly 2128 with reduced complexity.The use of LDO 21220 also allows infrared sensor assembly 2128 tooperate in a consistent manner, even if the input voltage from powersource 21230 changes (e.g., if the input voltage increases or decreasesas a result of charging or discharging a battery or other type of deviceused for power source 21230).

The various components of infrared sensor assembly 2128 shown in FIG. 23may also be implemented to operate at lower voltages than conventionaldevices. For example, as discussed, LDO 21220 may be implemented toprovide a low voltage (e.g., approximately 2.5 volts). This contrastswith the multiple higher voltages typically used to power conventionalFPAs, such as: approximately 3.3 volts to approximately 5 volts used topower digital circuitry; approximately 3.3 volts used to power analogcircuitry; and approximately 9 volts to approximately 11 volts used topower loads. Also, in some embodiments, the use of LDO 21220 may reduceor eliminate the need for a separate negative reference voltage to beprovided to infrared sensor assembly 2128.

Additional aspects of the low voltage operation of infrared sensorassembly 2128 may be further understood with reference to FIG. 24. FIG.24 illustrates a circuit diagram of a portion of infrared sensorassembly 2128 of FIG. 23 in accordance with an embodiment of thedisclosure. In particular, FIG. 24 illustrates additional components ofbias correction circuitry 21212 (e.g., components 21326, 21330, 21332,21334, 21336, 21338, and 21341) connected to LDO 21220 and infraredsensors 2132. For example, bias correction circuitry 21212 may be usedto compensate for temperature-dependent changes in bias voltages inaccordance with an embodiment of the present disclosure. The operationof such additional components may be further understood with referenceto similar components identified in U.S. Pat. No. 7,679,048 issued Mar.16, 2010 which is hereby incorporated by reference in its entirety.Infrared sensor assembly 2128 may also be implemented in accordance withthe various components identified in U.S. Pat. No. 6,812,465 issued Nov.2, 2004 which is hereby incorporated by reference in its entirety.

In various embodiments, some or all of the bias correction circuitry21212 may be implemented on a global array basis as shown in FIG. 24(e.g., used for all infrared sensors 2132 collectively in an array). Inother embodiments, some or all of the bias correction circuitry 21212may be implemented an individual sensor basis (e.g., entirely orpartially duplicated for each infrared sensor 2132). In someembodiments, bias correction circuitry 21212 and other components ofFIG. 24 may be implemented as part of ROIC 21202.

As shown in FIG. 24, LDO 21220 provides a load voltage Vload to biascorrection circuitry 21212 along one of supply lines 21222. Asdiscussed, in some embodiments, Vload may be approximately 2.5 voltswhich contrasts with larger voltages of approximately 9 volts toapproximately 11 volts that may be used as load voltages in conventionalinfrared imaging devices.

Based on Vload, bias correction circuitry 21212 provides a sensor biasvoltage Vbolo at a node 21360. Vbolo may be distributed to one or moreinfrared sensors 2132 through appropriate switching circuitry 21370(e.g., represented by broken lines in FIG. 24). In some examples,switching circuitry 21370 may be implemented in accordance withappropriate components identified in U.S. Pat. Nos. 6,812,465 and7,679,048 previously referenced herein.

Each infrared sensor 2132 includes a node 21350 which receives Vbolothrough switching circuitry 21370, and another node 21352 which may beconnected to ground, a substrate, and/or a negative reference voltage.In some embodiments, the voltage at node 21360 may be substantially thesame as Vbolo provided at nodes 21350. In other embodiments, the voltageat node 21360 may be adjusted to compensate for possible voltage dropsassociated with switching circuitry 21370 and/or other factors.

Vbolo may be implemented with lower voltages than are typically used forconventional infrared sensor biasing. In one embodiment, Vbolo may be ina range of approximately 0.2 volts to approximately 0.7 volts. Inanother embodiment, Vbolo may be in a range of approximately 0.4 voltsto approximately 0.6 volts. In another embodiment, Vbolo may beapproximately 0.5 volts. In contrast, conventional infrared sensorstypically use bias voltages of approximately 1 volt.

The use of a lower bias voltage for infrared sensors 2132 in accordancewith the present disclosure permits infrared sensor assembly 2128 toexhibit significantly reduced power consumption in comparison withconventional infrared imaging devices. In particular, the powerconsumption of each infrared sensor 2132 is reduced by the square of thebias voltage. As a result, a reduction from, for example, 1.0 volt to0.5 volts provides a significant reduction in power, especially whenapplied to many infrared sensors 2132 in an infrared sensor array. Thisreduction in power may also result in reduced self-heating of infraredsensor assembly 2128.

In accordance with additional embodiments of the present disclosure,various techniques are provided for reducing the effects of noise inimage frames provided by infrared imaging devices operating at lowvoltages. In this regard, when infrared sensor assembly 2128 is operatedwith low voltages as described, noise, self-heating, and/or otherphenomena may, if uncorrected, become more pronounced in image framesprovided by infrared sensor assembly 2128.

For example, referring to FIG. 24, when LDO 21220 maintains Vload at alow voltage in the manner described herein, Vbolo will also bemaintained at its corresponding low voltage and the relative size of itsoutput signals may be reduced. As a result, noise, self-heating, and/orother phenomena may have a greater effect on the smaller output signalsread out from infrared sensors 2132, resulting in variations (e.g.,errors) in the output signals. If uncorrected, these variations may beexhibited as noise in the image frames. Moreover, although low voltageoperation may reduce the overall amount of certain phenomena (e.g.,self-heating), the smaller output signals may permit the remaining errorsources (e.g., residual self-heating) to have a disproportionate effecton the output signals during low voltage operation.

To compensate for such phenomena, infrared sensor assembly 2128,infrared imaging module 2100, and/or host device 2102 may be implementedwith various array sizes, frame rates, and/or frame averagingtechniques. For example, as discussed, a variety of different arraysizes are contemplated for infrared sensors 2132. In some embodiments,infrared sensors 2132 may be implemented with array sizes ranging from32 by 32 to 160 by 120 infrared sensors 2132. Other example array sizesinclude 80 by 64, 80 by 60, 64 by 64, and 64 by 32. Any desired arraysize may be used.

Advantageously, when implemented with such relatively small array sizes,infrared sensor assembly 2128 may provide image frames at relativelyhigh frame rates without requiring significant changes to ROIC andrelated circuitry. For example, in some embodiments, frame rates mayrange from approximately 120 Hz to approximately 480 Hz.

In some embodiments, the array size and the frame rate may be scaledrelative to each other (e.g., in an inversely proportional manner orotherwise) such that larger arrays are implemented with lower framerates, and smaller arrays are implemented with higher frame rates. Forexample, in one embodiment, an array of 160 by 120 may provide a framerate of approximately 120 Hz. In another embodiment, an array of 80 by60 may provide a correspondingly higher frame rate of approximately 240Hz. Other frame rates are also contemplated.

By scaling the array size and the frame rate relative to each other, theparticular readout timing of rows and/or columns of the FPA may remainconsistent, regardless of the actual FPA size or frame rate. In oneembodiment, the readout timing may be approximately 63 microseconds perrow or column.

As previously discussed with regard to FIG. 19, the image framescaptured by infrared sensors 2132 may be provided to a frame averager2804 that integrates multiple image frames to provide image frames 2802(e.g., processed image frames) with a lower frame rate (e.g.,approximately 30 Hz, approximately 60 Hz, or other frame rates) and withan improved signal to noise ratio. In particular, by averaging the highframe rate image frames provided by a relatively small FPA, image noiseattributable to low voltage operation may be effectively averaged outand/or substantially reduced in image frames 2802. Accordingly, infraredsensor assembly 2128 may be operated at relatively low voltages providedby LDO 21220 as discussed without experiencing additional noise andrelated side effects in the resulting image frames 2802 after processingby frame averager 2804.

Other embodiments are also contemplated. For example, although a singlearray of infrared sensors 2132 is illustrated, it is contemplated thatmultiple such arrays may be used together to provide higher resolutionimage frames (e.g., a scene may be imaged across multiple such arrays).Such arrays may be provided in multiple infrared sensor assemblies 2128and/or provided in the same infrared sensor assembly 2128. Each sucharray may be operated at low voltages as described, and also may beprovided with associated ROIC circuitry such that each array may stillbe operated at a relatively high frame rate. The high frame rate imageframes provided by such arrays may be averaged by shared or dedicatedframe averagers 2804 to reduce and/or eliminate noise associated withlow voltage operation. As a result, high resolution infrared images maybe obtained while still operating at low voltages.

In various embodiments, infrared sensor assembly 2128 may be implementedwith appropriate dimensions to permit infrared imaging module 2100 to beused with a small form factor socket 2104, such as a socket used formobile devices. For example, in some embodiments, infrared sensorassembly 2128 may be implemented with a chip size in a range ofapproximately 4.0 mm by approximately 4.0 mm to approximately 5.5 mm byapproximately 5.5 mm (e.g., approximately 4.0 mm by approximately 5.5 mmin one example). Infrared sensor assembly 2128 may be implemented withsuch sizes or other appropriate sizes to permit use with socket 2104implemented with various sizes such as: 8.5 mm by 8.5 mm, 8.5 mm by 5.9mm, 6.0 mm by 6.0 mm, 5.5 mm by 5.5 mm, 4.5 mm by 4.5 mm, and/or othersocket sizes such as, for example, those identified in Table 1 of U.S.Provisional Patent Application No. 61/495,873 filed Jun. 10, 2011incorporated herein by reference in its entirety.

Where applicable, various embodiments provided by the present disclosurecan be implemented using hardware, software, or combinations of hardwareand software. Also where applicable, the various hardware componentsand/or software components set forth herein can be combined intocomposite components comprising software, hardware, and/or both withoutdeparting from the spirit of the present disclosure. Where applicable,the various hardware components and/or software components set forthherein can be separated into sub-components comprising software,hardware, or both without departing from the spirit of the presentdisclosure. In addition, where applicable, it is contemplated thatsoftware components can be implemented as hardware components, andvice-versa.

Software in accordance with the present disclosure, such asnon-transitory instructions, program code, and/or data, can be stored onone or more non-transitory machine readable mediums. It is alsocontemplated that software identified herein can be implemented usingone or more general purpose or specific purpose computers and/orcomputer systems, networked and/or otherwise. Where applicable, theordering of various steps described herein can be changed, combined intocomposite steps, and/or separated into sub-steps to provide featuresdescribed herein.

The foregoing disclosure is not intended to limit the present inventionto the precise forms or particular fields of use disclosed. It iscontemplated that various alternate embodiments and/or modifications tothe present invention, whether explicitly described or implied herein,are possible in light of the disclosure.

Embodiments described above illustrate but do not limit the invention.It should also be understood that numerous modifications and variationsare possible in accordance with the principles of the invention.Accordingly, the scope of the invention is defined only by the followingclaims.

What is claimed is:
 1. An imaging system comprising: a system housing;and an imager array disposed in the system housing and adapted to imagea scene, wherein the imager array comprises a plurality of infraredimaging modules, wherein each infrared imaging module comprises: amodule housing, an optical element fixed relative to the module housingand adapted to receive infrared radiation from the scene, and aplurality of infrared sensors in a focal plane array (FPA) adapted tocapture an image of the scene based on the infrared radiation receivedthrough the optical element.
 2. The imaging system of claim 1, whereinthe infrared imaging modules are arranged within the system housing in amulti-level staggered configuration to increase a packing ratio of theimager array.
 3. The imaging system of claim 1, wherein: a first one ofthe infrared imaging modules comprises a first filter adapted to limitthe infrared radiation received by the first infrared imaging module toa first spectral range; and a second one of the infrared imaging modulescomprises a second filter adapted to limit the infrared radiationreceived by the second infrared imaging module to a second spectralrange.
 4. The imaging system of claim 1, wherein: at least a first oneof the infrared imaging modules is greater in size than a second one ofthe infrared imaging modules; and the first infrared imaging module isadapted to capture an image of the scene with a higher resolution and anarrower field of view than the second infrared imaging module.
 5. Theimaging system of claim 1, wherein at least one of the infrared imagingmodules is adapted to block infrared radiation outside a desired fieldof view of its infrared sensors.
 6. The imaging system of claim 1,wherein at least two of the infrared imaging modules exhibit parallaxrelative to each other and are adapted to provide stereo images of thescene.
 7. The imaging system of claim 1, wherein: each infrared imagingmodule comprises an optical axis extending from its infrared sensorsthrough its optical element; the optical axes of at least two of theinfrared imaging modules exhibit phase shifts relative to each other;images provided by the at least two infrared imaging modules are phaseshifted relative to each other; and the imaging system further comprisesa processor adapted to perform super resolution processing of the phaseshifted images to provide a result image of the scene with a resolutiongreater than the resolution of the at least two infrared imagingmodules.
 8. The imaging system of claim 1, wherein: the imager array isadapted to capture a first image of the scene using at least a first oneof the infrared imaging modules; the imager array is adapted to capturea second image of the scene using at least a second one of the infraredimaging modules substantially simultaneously with the first image,wherein the second image has a higher resolution than the first image;and the imaging system further comprises: a database comprising amapping from the first image to the second image, and a processoradapted to: map a third image provided by the imager array to the firstimage, wherein the third image has a resolution substantially equal tothe first image, and replace the third image with the second image toprovide a high resolution result image.
 9. The imaging system of claim1, wherein: one of the infrared sensors in a first one of the infraredimaging modules is defective; and the imaging system further comprises aprocessor adapted to replace image data associated with the defectiveinfrared sensor with image data associated with a corresponding one ofthe infrared sensors in a second one of the infrared imaging modules.10. The imaging system of claim 1, further comprising a processoradapted to: map the infrared sensors of at least two of the infraredimaging modules to a grid comprising a plurality of pixels; accumulatesignals from the at least two infrared imaging modules for each of thegrid pixels; and provide a result image of the scene from the gridpixels, wherein the result image has a signal to noise ratio higher thansignal to noise ratios of the images captured by the infrared imagingmodules.
 11. The imaging system of claim 1, wherein the imager arrayfurther comprises: a plurality of visible light sensor arrays, whereineach visible light sensor array comprises: a lens adapted to receivevisible light radiation from the scene; and a plurality of visible lightsensors adapted to capture an image of the scene based on the visiblelight radiation received through the lens.
 12. The imaging system ofclaim 1, wherein: a first one of the infrared imaging modules is adaptedto capture a first image of a first wavelength range of the infraredradiation; a second one of the infrared imaging modules is adapted tocapture a second image of a second wavelength range of the infraredradiation; the second wavelength range is a subset of the firstwavelength range and substantially corresponds to an absorption band ofa gas; and the imaging system further comprises a processor adapted tocompare the first and second images to determine a presence of the gasin the scene.
 13. The imaging system of claim 12, wherein: a third oneof the infrared imaging modules is adapted to capture a third image of athird wavelength range of the infrared radiation; the third wavelengthrange substantially corresponds to another absorption band of the gas;and the processor is adapted to compare the first or second images tothe third image to determine the presence of the gas in the scene. 14.The imaging system of claim 1, wherein each infrared imaging modulefurther comprises a processor adapted to: receive a plurality of pixelvalues associated with the images captured by the infrared sensors ofthe infrared imaging modules; map the infrared sensors to a coordinatespace, wherein at least one infrared sensor of each infrared imagingmodule is mapped to each coordinate of the coordinate space; andcalculate an offset correction term for each infrared sensor based onthe pixel values of all infrared sensors mapped to the same coordinate.15. The imaging system of claim 1, wherein the FPA of each infraredimaging module comprises an array of microbolometers adapted to receivea bias voltage selected from a range of approximately 0.2 volts toapproximately 0.7 volts.
 16. The imaging system of claim 15, wherein: asize of the array of microbolometers is approximately 80 by 60; and eachinfrared imaging module is adapted to be inserted into a socket havingdimensions less than approximately 8.5 mm by 8.5 mm.
 17. The imagingsystem of claim 1, wherein: the images are unblurred infrared images ofthe scene; at least one of the infrared imaging modules is configured tocapture an intentionally blurred infrared image of the scene; and theimaging system further comprises a processor adapted to determine aplurality of non-uniform correction (NUC) terms based on theintentionally blurred infrared image and apply the NUC terms to acorresponding unblurred infrared image to remove noise from theunblurred infrared image.
 18. A method of imaging, the methodcomprising: receiving infrared radiation from a scene at an imager arraydisposed in a system housing of an imaging system, wherein the imagerarray comprises a plurality of infrared imaging modules, wherein eachinfrared imaging module comprises a module housing, an optical elementfixed relative to the module housing and adapted to receive the infraredradiation from the scene, and a plurality of infrared sensors in a focalplane array (FPA) adapted to capture an image of the scene based on theinfrared radiation received through the optical element; and capturing aplurality of images of the scene substantially simultaneously using theinfrared sensors of the infrared imaging modules.
 19. The method ofclaim 18, wherein the infrared imaging modules are arranged within thesystem housing in a multi-level staggered configuration to increase apacking ratio of the imager array.
 20. The method of claim 18, furthercomprising: filtering the infrared radiation received by a first one ofthe infrared imaging modules to a first spectral range; and filteringthe infrared radiation received by a second one of the infrared imagingmodules to a second spectral range.
 21. The method of claim 18, wherein:at least a first one of the infrared imaging modules is greater in sizethan a second one of the infrared imaging modules; and the imagecaptured by the first infrared imaging module has a higher resolutionand a narrower field of view than the image captured by the secondinfrared imaging modules.
 22. The method of claim 18, wherein at leastone of the infrared imaging modules is adapted to block infraredradiation outside a desired field of view of its infrared sensors. 23.The method of claim 18, wherein at least two of the infrared imagingmodules exhibit parallax relative to each other and their capturedimages are stereo images of the scene.
 24. The method of claim 18,wherein: each infrared imaging module comprises an optical axisextending from its infrared sensors through its optical element; theoptical axes of at least two of the infrared imaging modules exhibitphase shifts relative to each other; images provided by the at least twoinfrared imaging modules are phase shifted relative to each other; andthe method further comprises performing super resolution processing ofthe phase shifted images to provide a result image of the scene with aresolution greater than the resolution of the at least two infraredimaging modules.
 25. The method of claim 18, wherein the capturingcomprises: capturing a first image of the scene using at least a firstone of the infrared imaging modules; capturing a second image of thescene using at least a second one of the infrared imaging modulessubstantially simultaneously with the first image, wherein the secondimage has a higher resolution than the first image; mapping the firstimage to the second image; mapping a third image provided by the imagerarray to the first image, wherein the third image has a resolutionsubstantially equal to the first image; and replacing the third imagewith the second image to provide a high resolution result image.
 26. Themethod of claim 18, wherein: one of the infrared sensors in a first oneof the infrared imaging modules is defective; and the method furthercomprises replacing image data associated with the defective infraredsensor with image data associated with a corresponding one of theinfrared sensors in a second one of the infrared imaging modules. 27.The method of claim 18, further comprising: mapping the infrared sensorsof at least two of the infrared imaging modules to a grid comprising aplurality of pixels; accumulating signals from the at least two infraredimaging modules for each of the grid pixels; and providing a resultimage of the scene from the grid pixels, wherein the result image has asignal to noise ratio higher than signal to noise ratios of the imagescaptured by the infrared imaging modules.
 28. The method of claim 18,wherein the imager array further comprises a plurality of visible lightsensor arrays, wherein each visible light sensor array comprises a lensadapted to receive visible light radiation from the scene and aplurality of visible light sensors adapted to capture an image of thescene based on the visible light radiation received through the lens.29. The method of claim 18, further comprising: capturing a first imageof a first wavelength range of the infrared radiation; capturing asecond image of a second wavelength range of the infrared radiation,wherein the second wavelength range is a subset of the first wavelengthrange and substantially corresponds to an absorption band of a gas; andcomparing the first and second images to determine a presence of the gasin the scene.
 30. The method of claim 29, further comprising: capturinga third image of a third wavelength range of the infrared radiation,wherein the third wavelength range substantially corresponds to anotherabsorption band of the gas; and comparing the first or second images tothe third image to determine the presence of the gas in the scene. 31.The method of claim 18, further comprising: receiving a plurality ofpixel values associated with the images captured by the infrared sensorsof the infrared imaging modules; mapping the infrared sensors to acoordinate space, wherein at least one infrared sensor of each infraredimaging module is mapped to each coordinate of the coordinate space; andcalculating an offset correction term for each infrared sensor based onthe pixel values of all infrared sensors mapped to the same coordinate.32. The method of claim 18, wherein the FPA of each infrared imagingmodule comprises an array of microbolometers adapted to receive a biasvoltage selected from a range of approximately 0.2 volts toapproximately 0.7 volts.
 33. The method of claim 32, wherein: a size ofthe array of microbolometers is approximately 80 by 60; and eachinfrared imaging module is adapted to be inserted into a socket havingdimensions less than approximately 8.5 mm by 8.5 mm.
 34. The method ofclaim 18, wherein the images are unblurred infrared images of the scene,the method further comprising: capturing, by at least one of theinfrared imaging modules, an intentionally blurred infrared image of thescene; and determining a plurality of non-uniform correction (NUC) termsbased on the intentionally blurred infrared image and applying the NUCterms to a corresponding unblurred infrared image to remove noise fromthe unblurred infrared image.